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ABSTRACT 

The Conference sought to provide a user-based and 
-oriented assessment of performance contracting for education. It 
focused on delineating the current interest in performance 
contracting, the technique's demonstrated strengths and weaknesses, 
and its potential as an alternative approach to school system 
resource allocation. Six commissioned papers in this report discuss 
respectively, major problems in performance contracting, the recent 
Rand-HEW evaluation of performance contracting, measurement and 
evaluation difficulties, a model for comparing costs of different 
performance contracting programs, more comprehensive measurement of 
educational outputs, and employment relations under performance 
contracting. Discussions of these papers and of administrators' 
experiences emphasized the many problems of measurement and 
evaluation that have emerged in performance contracting. The papers 
and proceedings form the basis for several conclusions, including: 

(1) evaluations of performance contracting must be based on more than 
the measurement of student achievement gains; (2) meaningful 
measurement of student achievement requires multiyear testing of 
group gains; (3) performance contracting is so innovative that 
reliable judgments of its utility cannot be made for several years. 
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ABSTRACT 



The National Conference on Performance Contracting sought to pro- 
vide a uaer-based and -oriented assessment of performance contracting 
for education. It focused on delineating the current Interest in per- 
formance contracting, the technique's demonstrated strengths and weak- 
nesses, and its potential as an alternative approach to school system 
resource allocation. Six conanlssloned papers discussed, respectively, 
major problems in performance contracting, the recent Rand-HEW evalua- 
tion of performance contracting, measurement and evaluation difficul- 
ties, a model for comparing costs of different performance contracting 
programs, more comprehensive measurement of educational outputs, and 
employment relations under performance contracting. Discussions of 
these papers and of administrators' experiences emphasized the many 
problems of measurement and evaluation that have emerged In performance 
contracting. The papers and proceedings form the basis fov several 
conclusions, including: 

1. evaluations of performance contracting must be based on more 
than the measurement of student achievement gains, for such gains are 
only one of several objectives or Important effects of the technique; 

2. particularly In evaluating changes In studei.c achievement, 
first-year measurement of individual gain scores Is inappropriate; more 
meaningful measurement requires multi-year testing of group gains; 

3. performance contracting Is so innovative that reliable judg- 
ments of Its utility cannot be made for several years. 
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I. liUroduction 



Ihls conference was the renult of two circumstances. The first was 
the great growth of performance contracting for education between the 1969- 
1970 school year and the 1970-71 school year. In 1969-1970, only one nation- 
ally known perfoimance contract had been let by a public school system; In 
1970-71, more than 150 such contracts were let. At least four reasons can 
be cited for the swift expansion of performance contracting. 

First, It seemed to hold sone answers to the persistent problem of 
effectiveness, especially for programs designed to alleviate the specific 
educational deficiencies of the dlsadv.iritaged. By linking payment directly 
and sensitively to results, performance contracting was supposed to moti- 
vate the private sector to realize fully Its presumably great potential for 
producing significant educational changes. 

Secondly, performance contracting seemed to insure more rational, ef- 
ficient resource allocation, for it was supposed to rely explicitly on 
measured outcomes. A school system that lets such a contract lo supposed 
to be buying tangible progress and paying only for value received. Moreover, 
certain performance contracts tried to build in future efficiency by speci- 
fying that the contractor use only cost-effective (rather than labor- 
intensive) methods chat the school system later could adapt for proprietary 
use. 

Thirdly, by involving the private sector in difficult, risky enterprises, 
performance contracts presumably encouraged the introduction and testing of 
the high, highly productive technologies that school systems need but have 
been unable to use and to integrate with their curricula. These technologies 
are supposed to produce long-term benefits when they are transferred from 
the contractor to the school system and fully incorporated during the final 
"transfer" or "turnkey" stage of the performance contract. 

Finally, performance contracting was seen both as a genuine response to 
Increasing community and governmental demands for palpable educational re- 
sults and as a possibly effective way to counter greving community resistance 
to approving ever larger appropriations. When a school system requests funds 
for a performance contract, in effect, it was thought to show the community 
in more precise ways what it can buy rather than asking it to contribute to 
the general, undifferentiated support of a school system whose "product" is 
unclear. 

Though these reasons seemed to have been compelling ones for Implementing 
performance contracting, no individual or group had yet provided a comprehen- 
sive, user-based and -oriented assessment of performance contracting. The 
conference sponsors felt strongly that such an assessment should be available 
it school systems were to continue to let performance contracts after 1970-71. 

The second circumstance which lay behind this conference was that sig- 
nificant experience with and evaluations of performance contracting had begun 
to emerge. 

The conference was organized to coincide with first publication of the 
results of the Rand-HEW study of performance contracting and took place 
shortly before publication of the results of the OEO experiment in performance 
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contracting. Rcprcscntatl ves of both Rand and OEO were conference partici- 
pants, so the conference was able to make use of data generated by the only 
two broad-Hcalc evaluations of performance contracting currently available. 

In general, the conference sought to answer questions relating to five 
particular aspects of performance contracting: 

I • Its origins 

Performance contracting was e response to specific financial, managerial 
and product needs of Industry and government — particularly those of the 
Department of Defense. To develop a functional conception of performance con- 
tracting, we must understand these roots. In addition, we must clarify the 
educational needs which performance contracting is supposed to meet, bring 
out any important differences between these needs and those of government 
.and industry, and dctcrmlpc W *-her the technique has undergone any signif- 
icant alterations intended to make it more suitable for its new application. 

In short, we must determine why and how educational performance contracting 
differs from earlier fornw. Given Its roots, what can we expect of perform- 
;ince contracting In education (.and what must we not expect)? For example, 
can we Justify treating progress on a standardized test as a "product" — 
like a transistor or a gunsight — that cither meets or falls to meet con- 
tract specifications? Is this .a legitimate objective for a school system? 

2. Its structure 

What are the essential elements of a performance contracting program, 

.and how do they Interact? What alternative types of performance contracts are 
available to the school system? We must Identify those aspects of a perform- 
ance contract that can be varied with the needs of a particular case In order 
to Insure maximum benefits and to produce a true reflection of a school sys- 
tem's needs. This Identification, of course, will demand an understanding of 
the dynamics of performance contracting, of Its Internal logic. We must 
examine the strengths, flexibility, pitfalls, and comparative merits of vari- 
ous components and types of components. For example, what kinds of evaluation 
mcasure.s are available, and which ones should we use In a particular case? 

At what times during the life of the contract can we apply them, and can we 
gradually adapt them to any now output quantities that may emerge as the con- 
tract is executed? How should these measures be used to determine payment? 

Can tlicy be refined during the life of the contract without undermining the 
ontl re scheme of "payment for performance" that the contract embodies? 

3. Its specific applications 

Few school systems have tried to determine what circumstances favor let- 
ting a performance contract, aside from the general need for real progress In 
educating their "clients." Detailed consideration must be given to precon- 
ditions such as the legal and informal relations with government at all levels 
the types of learning problems to be solved, and the details of the process of 
school finance. We also must ask how well a performance contract suits these 
preconditions in contrast to the suitability of competing in-house modes such 
as "model teaching experiments" and other, more traditional schemes of program 
development, implementation, and evaluation. We must study the Intra-systemlc 
effects of performance contracting, its specific Interactions with system 
components such as teachers, organized labor, physical facilities, accounting 
practices, planning frameworks, parents, and Instructional activities that are 



noL included in the contract under consideration. For example) what kinds of 
problems must we expect in phasing "contracted students" back Into the o.'dlnc*^ 
curriculum, especially in light of the new ways of learning and different knowl- 
edge they may have .acquired? More gener.illy, how slall we determine a perform- 
,uice contract's relations with the goals and structure of the school system by 
defining the policy, managerial, and operatlon.sl responsibilities of both par- 
ties to the contract? 



4. Its results to date 

What does our experience with performance contracting for educatlor. say to 
the aforementioned aspects of structure and application? What unexpected prob- 
lems have emerged, if any? Are they inherent difficulties, or simply those of 
inexperience? Is there enough reliable data to generate a set of recommenda- 
tions? If so, what recommendations would be most useful to the school system? 

To the private contractor? Until now, the almost complete lack of relevant 
data has precluded .answering such questions; but at this point, experience can 
nswer questions .about such topics as the relationship between contractors and 
school system teachers and the criteria used to select "contracted pupils" 
from a larger target group. School administrators now can (and should) h.avc 
on .awareness of and reaction to the demonstrated strengths and weaknesses of 
performance contracting. 

5. Its future 

In light of all the preceding considerations, where can performance con- 
tracting go, and where do users want 1 t to go. School administrators must iden- 
tify weaknesses that should be alleviated and strengths that should be exploited. 
Finally, they must determine what place they envision for perform.-ince contract- 
ing in the larger context of educational planning. 

In more operational terms, the conference sponsors felt that answering 
this broad variety of questions would require achievement of four objectives: 

1. To cull and analyze relevant experience in performance contracting 
for education; 

2. To identify and respond ro the salient issues, which principally 
relate to four areas: 

A) Measurement and Evaluation. 

B) Articulation of the contractor's programs with the school sys- 
tem's and with elements of the school system's environment. 

C) Application of performance contracting to various types of pro- 
grams — remedial, skill-training, research oriented; and exter- 
nally (federally) funded programs, as well as those that cut 
across grade and subject lines. 

D) Specifications of what constitutes a "good" performance contract. 

3. To promote the practitioner's awasness and understanding of the issues 
that currently concern both researchers and administrators; and 

4. To identify the major features of a performance contract and the cri- 
teria used to select it, as well as to show how the contract and these 
criteria can best articulate school system objectives. 
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Mclhodology 



To nchlovc the general goals and particular objectives of the confer- 
(Mue, Its Bponsors felt that they would have to engage in a certain basic 
Hot of activities. The principal members of this set included: 

” Jcl ineating the dimensions of current Interest in performance 
contracting, the reasons for it, and the particular school 
system needs to which performance contracting responds; 

® summarizing strengths and weaknesses of performance contract- 
ing as its users have perceived them and, when possible, 
evaluating these pcrceotions against subsequent performance; 

® defining the roles of third parties and the extent to which 
they can heln school administrators plan, manage, and evalu- 
ate a oerf ormance contracting program; 

” projecting the imnlications of pcrfonnance contracting for 
current and future school system practices and organizations; 

assessing the utility of performance contracting as an alter- 
native mode of resource allocation and making recommendations 
useful to those nlanning to engage in a nerf ormance contract. 

The confeicncc sponsors — the American Educational Research Associa- 
tion and the American Association of School System Administrators — felt 
that the best way to carry our conference activities was to assemble a 
diverse groun of researchers and administrators, all of whom would have 
significant experience with performance contracting and related areas such 
as the measurement of achievement gains. As Appendix A indicates, this 
group included: 

^ an acknowledged specialist in the development and implementa- 
tion of performance contracting programs; 

® representatives from the two Federal agencies moat deeply 
involved with performance contracting (OEO and USOR) ; 

® principal investigators for the Rand-HEW study of performance 
contracting; 

^ researchers and specialists in a variety of fields crucial to 
performance contracting — educational cost-effectiveness, 
broader planning frameworks such as Plannlng-*Programroing-Budget- 
ing systems, measurement and evaluation both of student achieve- 
ment and of program performance, school system "politics," 
curriculum development, and the nrocess of learning; 
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Find LriK^ 



The findlngB of the conference comprl -rd both tin coininl Hsloned papers 
and the proceedings (discussion and suininary) . The flrh^ part of this section 
of the final report duplicates in full the commissioned papers. These papers 
dealt with six different aspects of performance contracting. 

Major Frohlema In Performance Contracting for Education; An Tntroduc^tlon , 
l)y Donald M. I.evlnc, discusses: 

1. measurement, testing, and evaluation problems related to such areas 
as uncertain affective outcomes, proxy measures, reliable measurement 
of cognitive outcomes, sub-optimization of the learning program, and 
learning gain persistence; 

?. resource measurement and assessment problems primarily related to the 
difficulty of discerning the impact of a broad range of inputs upon 
learning program outcomes; 

3. situational conditions, such as the curricular locus of the performance 
contract, the characteristics of the student population that will partici- 
pate, teacher and union relations, school system flexibility, and school 
system environment. 

The conluslon of this paper emphasizes not only the aforementioned areas of 
concern, but also the social impact of performance contracting and Its long 
term implications for school systems. 

Polly Carpenter's An Kvaluatlon of Performance Contracting for HEW sum- 
marlzes the findings of the Rand performance contracting study team with 
rej;ard to the efficacy of perfornance contracting as a technique for 
solving problens in education. It does rot iltec:pt to compare specific 
programs because of the wide diversity of conditions In which programs 
were inplenented and because of the diversity within the programs 
theiaselves. Ratiter, the effects of perfornance contracting are discussed 
in six areas of eduent Iona I concern: Instruction, student learning, evaluat ion, 

program uanagenent, program cost, and learning system contractors. The major 
disadvantages and advantages of perfornanct* contracting as a technique 
also arc noted. 

The third paper, Some Problems In As s essing Educational Performance , 
by Robert Ebcl, discusses eight neasureuent and evaluation issues: 

1. The problem of what to assess: performance or capability? Cognitive 

or non-cognitive learnings? Knowledge or intellectual skills? 

Attitudes toward and satisfactions in learning? 



-6- -X 



2. The problem of how lo assess it: iiu’ormal observation or formal 

testing;? Objective or essay tests? I'lorm-ref eronced or criterion 
referenced tests? Familiar or novel tests? Motivating the 
examinee . 

'i. The i)roblen of too much Lesting: values and drawbacks of frequeivt 

testing, Need for data vs. time devoted to testing. 

4. The problem of who should assess it: values and limitations of 

assessment by contractor, client, and outside evaluator, the 
students . 

5. The problem of obtaining the test; teat publisher? Custom 
made? Test specifications, item writing, pre-testing, test authors. 

6. The problem of test security: useful and harmful teaching to 

the test, purloined items or tests and coaching, equivalent new forms. 

7. ,The problem of score interpretation: who is responsible, item 

proportions or test scores, individual or group scores, sampling 
errors, gain scores, adjustments for ability. 

8. The problem of determining acceptable performance: mastery, 

minimal p’*of icicncy , a priori judgment or ^ pos teriori data, 
individual differences. 

This paper concludes with recommendations for the future. 

Program Cost Analysis in Educational Planning , by Sue A. Haggart, 
describes a planning cost model and its use in estimating program cost. 

The model explicitly deals with the problem inherent in using an 
undefined cost per student as an input for comparing programs. 

The model is used to systematically and consistently generate two 
types of program cost: 

° The comparable replication cost of the program, 
riie cost of the program in a specific district. 

The comparable replication cost , (CRC) is the output of the model that 
attempt.* to put the dollar costs of a program in different school 
districts throughout the country on a comparable basis . This means that 
allowances are made for price differentials and for differences in the 
use of existing (or surplus) resources available within specific districts. 
The CRC is, therefore, essentially an index or synthetic number to be 
used only in making interdistrict program cost-effectiveness comparisons. 



TliL* nod ul Is used Lo ost. inaLe the cost of the program in a specific 
distri('t by first dotermining the incrumentai resources required based 
()H tlu‘ availal)ili ty of resources within the district, and then using 
district-specific prices for the resources. The output of the model, 
used in this mode, is the estimated cost of a program or alternative 
programs for the district. 

Selma J. Mushkin*s Performance Toward What Result; An Examination of 
S ome of the Problems in Outcome Measureraent , reviews the approach and 
and tentative findings of a study ongoing at the Public Service Laboratory 
of Georgetown University on statistical measurements of educational outputs. 

It presents work that has been done on (a) standardization of achievement 
testing for population characteristics (under the caption of the **SIR** 
adjusted educational achievement index), and (b) measurements of the 
child’s perception of self, and of society. The noncognitive measures 
importantly impact on the development of the child and have important 
consequences for learning, and this paper seeks to assess what can now 
be done about building affective testing into planning for performance 
payments • 

The findings of the Public Service Laboratory study are presented 
in summary form with emphasis on the question ”Uhat is the relevance 
of the study’s tentative findings to the question of performance?” 

Finally, Employment Relations Under Performance Contracts , by Myron 
Lieburnan discusses seven important aspects of employment relations under 
performance contracts : 

1. What are the actual employment relations under current 
])er forma nee contracts? 

2. What is the actual and potential impact of these relation- 
ships upon regular school district employment relations? 

3. What has been the response of teacher organizations to the specific 
arrangements providing for performance contracts? 

4. To what extent, if any, are perf orr.iance contracts blocked or modified 
by teacher attitudes and policies with respect to employment relations 
in performance contracts? 

5. To what extent do performance contracts provide for ’’merit pay,” 
and how have arrangements for merit pay worked out in practice? 

6. To what extent, if any, do teacher-board negotiations deal with 
performance contracts, e.g., do teachers try to prohibit the 
contracting out of educational work? 
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7. What appear to be the main issues and alternative soo.utions or 
I approaches concerning employment relations in performance 

^ contracting and how are they likely to affect the future of 

I performance contracting? 

The second pati of this section, which follows the papers, consists 
of excerpts from the transcript of the conference. Although the excerpts 
included here do not constitute an exhaustive catalogue of the ideas 
and conclusions voiced at the conference, they are typical of the 
interaction and analysis that characterized both the plenary and the small 
group discussion sessions. The conclusions of the next section of this 
report arc based largely on these discussions. 
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tlAJOR PROBLEMS IN PERFORMANCE CONTRACTING 
FOR EDUCATION; All INTRODUCTION 

Donald M. Levine 
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M_ni()j_l’j-c'l.k'riis ni rot-f()£nwiux'_CVMurnctina 
]''oT I I i- Mrioi 1 : ' An 'I i '.n o ji : cT V(Ti f 

Doimlci M. Levine 

Ontario IiiGiitutc 
for 

Studies in Education 



In trodu ction 

7'lie last ai'.adeniic year hrouglit witli it an unprecedented and almost 
unbalanced surge in the growth of pcrformr.nee contraeting for publie 
sehool system.?. In 1969-70, only two major performanee eontraets 
for education were fully operational; but in 1970-71, more than 100 were 
o{)erational, and many more were in the planning stage, The parti- 
cular reason.? foi- tliis growth are manifold, and we shall refer to them 
later, but in almost every ease, one of two attitudes seems likely to 

. ^ 

prevail: the participating school system either sees performanee eon- 

tracting as a self-sufficient remedy, as some si>ecies of panacea: or it 

justifies an eajfcrness to become involved with performance contracting 

under the heading of c;.\perimentation, which lowers the perceived "'isk 

of stich programs. As Dr, Gordon MacAndrew, Superintendent of School 

2 / 

in Gajy, Indiana, put it, "How can we lose?"—' In both cases there is 
a great temptation to plunge into performance contracting without due 

* Prepared for ilie National Conference on Pei formance Contracting for 
Education: spon.sored by rlie y\merican Iiclucational Research Association 
cind the y\merican Association of School y\dmini.strators through a grant, 
.from ilic Uniici.i Slates Office of Education. 

i/ 

J. P. Stucker ;m:l G. R. Hall, 7 he I’erformance Contracting Cioncept i;i 
Edu cati on , R/\N7h,Santa ivionica, Ca 1 iioj ii )a TTd Vir pp~^^^^^ 297 ' 

-'^Glob.- in Mai l , d'oronto, Canada, jN'ovonber 1 S, 1971, p. W-6. 
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considi'ration of that technique’s applicability to the school system in 
question. Even grantinp; the "no- penalty" nature of experimentation, 
this can be a serious error, fora poorly designed performance con- 
tracting program can prove financially dangerous, organizationally 
disruptive, and educationally destructive. It is vital, therefore, to 
examine problematic aspects of the performance contracting technique 
in order to determine its general feasibilit}' for education and then to 
explore tlie situational constraints on its applicability to particular 
school systems, constraints that will shape'our efforts to plan for a 
performance contracting program. The first two sections of this paper 
deal with issues connected, respectively, with the problems of measure- 
ment, testing, and evaluation and with the problems of resource mea- 
surement and impact assessment. The final and major section discusses 
striictiiral and dynamic school system characteristics that must be 
considered in planning for a performance contracting program and 
seeks to specify preconditions that seem to favor successful implemen- 
tation of a performance contracting program and to encourage school 
sy.sicm receptivity to .'^uch an approach. 

Because of our lack of long-term experience with performance 
contracting for education, some of our answers to these questions will 
be deductive rather than strictly empirical. Yet the theory of perfor- 
mance contracting is sufficiently explicit, and the need to answer these 
questions sufficiently pi'cssing, to justify some departure from the more 



..'■.8 



restrictive research viewpoint. 






O 

ERIC 



-3- 



Tcclinicnl rroblom s 

From the viewpoints of both resenrcli and practice, the most impor- 
tant issue in performance contracting for education may well be that of 
measurement, testing, and evaluation. It is important to understaiid 
the different purposes for whicli v/e evaluate performance, because 
those purposes affect the kinds of measures and tests we can use. First, 
and most obviously, we evaluate performance in order to determine 
contractor payments. The utility of evaluation for tliis purpose depends 
at lea.st as much on the payment schedule-which links performance 
measurement to payment- as it does on the actual mea.siircs and tests 
applied, for the schedule may be adjusted to offset suspected inequities 
and errors i.u the tools of measurement. Still, the validity of such tools 
is the basic problem that has attracted the most critical attention, and 
this conference hopes to examine major criticisms such as: 

1. The standardized tests generally used to determine at least part 
of the contractor's payment are frequently inappropriate to the contractor's 
program. Standardized tests designed to measure generalized learnings 
necessarily ignore the special content and objectives of a particular 
contracted program as well as other important circumstances affecting- 
program success, from variations in student ability to heterogeneity of 
test group population. And they are nor designed to measure the succes.s 
of relatively short term and special purpo.se programs, such as tho.se 
involved in most performance contracts. Moreover, such norm- 
referenced tests aj-c l)ased on imperfectly controlled populations which 
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may differ g)'c,*ai!y fi'om the siiidciit population lx;ing tested, wliose 
scores consequently suffer from cultural and, in some tests, ability bias 
unrelated to the learning in question. 

2. J’ho individual achievement test gain scores often used to compute 
payments do not reflect actual achievement only. As Sta.ke and Waldrop 
have put it, gain scores may "assure tlic apix^arance of learning where there 
is no learning at all" because "the conventional achievement test does not 
liave the necessaiy content validity for individual student assessment. 

The large proportion of error and low reliability of simple individual gain 
computed by taking the arithmetic difference of pre- and post- test scores 
make gain scores a dangerous basis for payments. Some of this danger 
can be allayed by penalii^ing the contractor for achievement losses to the 
same extent he is rewarded for gains, which will average out fortuitous 
over- and under- payments: by focussing on group gain scores, which 
should avciagc out errors before payment is made; or by mathematically 
dropping out a percentage of error in order to derive "actual" achievement 
scores. But each of these solutions has its own, mainly operational draw- 
backs. (Another, less frequently mentioned defect of traditional gain 
scores is that degree of gain correlates closely with the pre-test score, 

but this problem seems more amenable to an explicit, mathematical solu- 
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--Robert Id. Slake and James L. Waldrop, "Gain Score Errors in Performance 

Contvncti.t':’:, " n paper presented to the American Educational Rcsoarcli Associaiiou, 
P'elu'uary, G971. 

^/See Koben- A. I'clJme.s.ser, "'Jests and the ETS Role in Performance Contracting, 
a papoj- ii.i.'.’rnally cjj’culaicd at E'J’S. i c 
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3. The giTulo level increases inio which gain scores arc translaicd 
for payment and publication purposes have all the inadequacies of gain 
scores hut introduce more error because grade level increases measured 
on different type.s of test'j are not comparable. Thus, grade level in- 
creases imply a false degree of certainty, and their very simplification 
encourages explicit bur meaningless comparisons. Grade level increases 
may have pragmatic advantages, but they are a totally inadequate basis 
for comirensation or for comparative evaluations within and among 
performance contracting programs. 

4 . The criterion-referenced tests so frequently mentioned as substitutes 
for standardized achievement tests have their own defects. They are 
particularly open to contractor abuses in design and execution and there- 
for require veiy careful negotiation and suiaervision. More importantly, 
they encourage optimization of limited, myopic, or even irrelevant curricu- 
lar objectives at the exjx^nse of higher program and system goals. The 
critei.'ia of such tests are not necessarily germane to generally accepted 
notions of education. This problem might be circumvented if contracting 
school systems had meaningful goal and objective hierarchies to which they 
could related proposed criteria, but most systems are far from this 
degree of self-consciousness. 

This last criticism closely relates to a second purpose of evaluation-- . 
to assess the value of the jxirformancc contracting program within the 
school system., to go beyond achievement tests in order to isolate the pro- 
gram’s pcn'formance against such higher goals as loclinological innovation 
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ancl lo discern its varied impacts on system strticture and jx^rsonncl. This 
seems to he the kind of evaluation that tlic I^AND/IIEW study stresses, and 
it is vital to scliool systems at tliis experimental stage in the evolution of 
performance contracting. Clearly, such evaluation should he long-term 
since the real utility of performance contracting depends on its ability 
to effect lasting school system improvements. And it must employ a 
range of measures within which objective, formalized tests will occupy 
only a small place. Exactly what mix of measures should be used for 
program evaluation remains unclear, hut it must depend on the school 
system's particular objectives in deciding to implement performance 
contracting, and it should include a variety of behavioral measures in 
order to catch the profound effects of the new roles that this technique 
imposes on many people and groups within the system. Again, a hierarchy 
of goals and objectives seems almost indispensable for designing this kind of 
evaluation. 

The aforementioned purposes of evaluation are concerned witli output 
measurement of various kinds; but the third major purpose of evaluation 
focuses as much on inputs and this leads us into the second group of 
technical proldcms-- resource measurement and impact assessment. This third 
purpose is to discriminate among the contributions to program outcomes 
that are made by the diverse participants in and elements of a performance 
contracting program. The end of such discrimination is the development 
of the kinds of input-output models and cost/effectiveness relationships which 

r 
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will pei'iiiii us to rationally rcsirLiciurc pcrfoririancc contracting pro- 
grams and to allocate resources within and among competing progrtims. 

It is very difficult to discern the discrete impact of a certain input or 
resource level upon program ouiccmcs - again, because of outcome 
measurement difficulties. In this case, though, the ability of the 
standardi'/ed test to measure achievement gains reliably is some- 
what irrelevant, for the problem is that most standardized tests are 
combined measures which lump outcomes together, that do not de- 
compose behavior sufficiently for us to build meaningful models. This 
problem of combined measures seems technically irreducible in many 
cases, but it does not actually destroy our ability to deidve some idea 
of the effectiveness of such gross inputs as the technological aspects 
of the contractor's program or the affective impact of school system 
personnel. We can use measures other than standardized tests - structuied 
interxdews, clinical observation, questionnaires, etc. - to get some feel 
for the impact of these resources. 

Another problem arises when we come to measure the actual re- 
sources used in a performance contracting program and attempt to build 
cost-structures for purposes of comparison and decision-making. The 
resources consumed may have hidden or sunk costs, or their cost may 
vary significantly across the country. It can be difficult to separate costs 
to the contractor from costs to the school system, as it can be to deter- 
mine precisely what resources should and should not l)e accounted for. 
Clearly, this problem hinders cost/effectivcncss studies of performance 
contracting; but it seems more tractable than that of output measurement. 
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Pi eron Jitions 

One of ilic stnicturMl preronJi lions ihrit most influences planning for 
a pGi’forinancc contracting program is the citrricular l ocus of the problems 
whose solution night c;ill for a ix:,r forniance contracting program. Certain 
parts of a .scliool system's curriculum are more amenable to performance 
contracting than other parts, and for rathej‘ convincing reasons.-^ The 
"best” curriculum segment for this technique is vocational education, 
or any segment that primarily involves training activities. This is 
because training programs traditionally have objectives that can be 
accurately stated in concrete, beltavioral icrms. A reasonable objective 
of a typing course, for example, might be "to have each student typing 
at 40 words peyminuic. with less than 15 % ej’ror after twenty hours of 
instruction". Such an objective needs no major redefinition or clari- 
fication to serve the purposes of peid'ormance contracting- -it is suf- 
f)( ij' specific foj- both parties to know what task must be performed, 
and it is easy to derive unequivocal performance mcasuj'cs from this 
objective, thus elim’inating the possibility of serious te.sting--and hence 
remuncratlve--disioriion. And in cases lil;c this, the frequently 
dysfuncijonal p3'actice of "leaching to the test" is prcci.scly what the school 
system wants. 



l/S-jc Sincker and Hall, The Pcrfonnanco Cioiurac.iing Concept in lidu cati on, 

p. 11. 
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A much, clcsirnblc, but seemingly prnciicnblc cun-jcuhir locus 
is bnsic vcrlial .tikI qu.'iiU i mtivc .sKills. .Hcndln^; nncl ninth olijcctivcs 
arc seldom so complete and unequivocal ns to preclude debate between 
conirncioi- and school system about esacily what is to be achieved and 
how it is to be measured; and third parly supervision and/or ai’biiration 
often is necessary when the parties to the contract arc disputing ob- 
jectives and performance nicasin-cs for these basic programs. Yet 
both cducator.s and contiactors nave seemed willing to accept a range 
of surrogate or proxy measures (mainly connected with standardized 
tests) when negotiating for pi ograms at this level. Such acceptance 
raises serious and possibly expensive questions about the validity of 
the performance- payment schedule, about sub-optimization, and about 
the real meaning and persistence of verbal and mathematical acliievenic'.it.~ 
Still, it has provided a inuiually held, cperational set of objectives, 
measures, and tests for performance contracting. A major reason for 
this dcvclopnicni, according to our experience to date, is that the need 
for new, more effective approaches to learning basic verbal and maihc- 
inaiical skills has dictated that most performance contracts- -whether 
inicgjal O)- cxix'i i menial (dcnioi)siraijon)--conceiiiraic cxclu.sively on 
basic, reading and math ability. Though ilie.se curricular foci are less 




— ^If the coniraci riep;oiiaiors liave fairly accurate perci.-ptious of the 1' Kap.c 

between "jcal" peifonnaiice (what the .sy.sicm actually want.":) and mea.suros 

like "grade cciiiivaleiice", of com se, ilicy gave a heircj’ chance of producing 

an effeeiivc’ perrormanco eoniracl. But the defect.*: of grade eciuivaleiux: 
render such perec'pnoii.s r.ij'c. 






well yuili'd 10 ihc: clear definition and nieasuremcnt of [X^rformnnee that 
training progi'ams can provide, school administrators have been willing 
to accept pi-oxik'S for the sal;c of whatever solutions ixn*forinancc 
contracting can find. 

The ctirricular loci that seem least appropriate for a performance 
contract tne those with high normative O'’ affective content. It is hard 
to pinpoint these loci, since tj'aining programs and basic skills programs 
themselves have significant non- cognitive a.spects; but we might look 
to the curriculum as a whole to have more affective influence than any 
of its parts. Certainly, the gross curriculum and its more affective 
loci share the same impediment to performance contracting--highly 
debatable outputs that generally defy meaningful quantification. It is 
important to realize that the affective results of a performance contract 
for any curricular locus are questionable, and few efforts have yet been 
made to define or limit those results. Given the contractor's profit 
maximing behavior, which usually is channeled only by certain cognitive 
objectives and measures and constrained only by tina and resource 
specifications, a performance contracting program may well have serious, 
inforcsecn affective consequences.—'^ Moreover, such change can be 
unprcdictalde, and, from Lite school administrator's point of view, un- 
controllable. Twj of the three current performance contracts that 



I\ case in point is modifjc.ujon of student bchavior--and- achievement of contract 
objectives- -through an extrinsic reward system. 
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involvc <'in ciuirc cuiTiculuin seem to hnve recognized this difficulty, 
for iliey tie payment only to the reading aed matli pana of their programs. 

A second important precondition is the nature of the school system's 
student population, and the data tlie system has collected to describe 
that population. Data along several relevant dimensions- -past i^erformance 
ir school and on "diagnostic" tests, age, length of time within the school 
system, family background, etc. --may be used to write final contract 
specifications. Most contractors will insist on such information as a 
basis for rationally negotiating the limitations of enrollment in the con- 
tracted program (unless, of course, the program is school- wide). 

Such data are particularly important for explicitly experimental contracts, 
where control cf subject (student) characteristics can be essential to 
subsequent evaluation of the performance contracting program. 

It would be rash to contend that any particular type of student 
population inherently favors letting a performance contract, yet most 
performance contracts to date have involved groups of disadvantaged 
students. Thus we must ask v/hich characteristics of this kind of 
population seem to fit particularly well with. certain kinds of pc 2 ’formance 
contracts and explore the causes of the pj-cvalencc of disadvantaged 
students as the participants in performance contracting programs. ' 



i'^^Gary Indiana, and jacksonviDc, Florida. See J. P. Stuckcr and G. R. 

Mall, The Performance Contracting Concept in IMucniion, p. 31. 
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One obvious cause is that this is a population for which the school 
system's own methods frequently liave failed to produce lasting, signi- 
ficant achievement gains, or even to mai)itain previous rates of measured 
achievement growth. In such cases, outside help--especially help that 
is explicitly motivated to improve student performance--may be indicated 
as nothing more than a last resort. Secondly, disadvantaged students 
often seem to have relatively clear-cut problems, both of academic 
achievement and of more general behavior. Thus it seems easier to 
specify objectives for this population and to create fairly homogeneous 
target groups for operational, experimental, and evaluative purposes. 

A third apparent cause of the high incidence of disadvantaged students 
in performance contracting programs is that the problems of this group 
seem intimately connected with motivation, and performance contractors- - 
who come from the competitive world of business and operate in 
situations that involve highly motivating incentives- -have shown par- 
ticular strength in designing effective student motivation systems that 
include a range of extrinsic and intrinsic rewards. The affective by- 
products of such systems are, indeed, questionable, and they can entail 
painful issues of pedagogic ethics; yet from the limited viewpoint of 
effectivene.ss in cognitive development, they hold promise for the 
disadvantaged. Finally, there are powei'ful pragmatic considerations that 
apparently suit disadvantaged populations to performance contracts. 

Such populations have attracted inc.rea.sing attention in the community, 
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QR have school sysiem failures to improve their achievement scores. 
Hence any new program for the (.lisaclvantagcd, and especially one that 
promises concrete, short-term, highly visible performance gains, is 
politically attractive and constitutes a powerful community relations tool. 
Moreover, the presence of federal funds for the disadvantaged tends to 
encourage school systems to try unproved, experimental programs 
for that group, and the pei-formance contracting concept builds in the 
objective specification, cost structure, and evaluation that federal funding 
requires. And in general, many school systems view the use of per- 
formance contracting for the disadvantaged as a "no penalty" situation, 
since they feel that the contractor cannot aggravate and may ameliorate 
the pliglit of the disadvantaged. 

This analysis of current performance contracting applications cannot 
tell us that a student population witli significant numbers of the disad- 
vantaged is a necessary precondition for a performance contracting 
program. What it can imply is that positive indications for performance 
contracting implementation would include a student population or 
sub-group: 

a. that has failed to show satisfactory achievement gains in response 
to school system efforts; 

b. that has relatively specific problems and characteristics; 

c. that has apparently low achievement motivation. 
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A third prcconciition concerns the school sj'stcm's teachers and its 
interactions with ihcin. Organizational climate and cultin-e are important 
factors here, for teachers will respond to performance contracting 
largely in the ways the school system has taught them to respond to 
other innovations and external influences. If in the past the system has 
cultivated the teachers ' achievement motivation by creating a climate which 
stresses autonomy, professional responsibility, moderately high stan- 
dards, and specific rewards- -either intrinsic or extrinsic--for success, 
it is probable that teachers will liave developed the ability to exploit 
many factors in their cnvironm.ent for their own- -and presumably, their 
pupils '--success. In the best sense of the v/ord, they will be entrepre- 
neurs- -flexible, inventive, and adaptive. In such a case, the school 
administrator may have little need for a performance contracting program 
because his in-house levels of achievement are satisfactory; but if he 
docs perceive the need for such a program, he can have some assurance 
that his personnel will be able to participate creatively in the contractor's 
program and to take full advantage of it for their own purposes during the 
turnkey phase. (With a school system like this, the administrator also 
may want to explore the possibility of so-called "in-house" px^rformance 
contracting, which is more a control technique than a resource allocation 
system. ) l£spccial]y when tlie contract specifies involvement of school 
system teachers, as most do, achievement oriented per.sonnel are an 
excellent precondition for performance contracting; yet the administrator 
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must be careful about intrcxlucing the performance contracting concept 
and the contracted program. Unless lie stresses that performance 
contracting is Ixjiiig used primarily to introduce new technologies and 
strategies to help teachers improve their effectiveness; that teachers 
will be involved in planning and operating the performance contracting 
program; and that the contractor will work closely with teachers, his 
highly motivated personnel are likely to feel by- passed and unfairly 
excluded from decisions important enough to merit their participation. 

In contrast, a school system whose organizational climate fails to 
encourage achievement motivation because it emphasizes conformity, 
risk-aversion, and conflict- avoidance may expect to have teachers 
concerned primarily with accommodation to the bureaucracy and moti- 
vated by considerations of job-security. A system- teacher situation of 
this type is at once most likely to require such techniques as performance 
contracting in order to improve student achievement, and least likely 
to be able to implement performance contracting successfully. Although 
the norm of conformity and the habit of accommodation can elicit token 
compliance with the performance contracting program, the teachers 
are likely to view any such innovation ns a threat to job security, and 
hence to resist it staunchly. More importantly, they may be both unwilling 
and unable to participate effectively in the contracted program (thus 
aggravating their own perception of job- threat),’ and will tend to welcome 
they turn- key pha.se not as an opportunity to acquire a proven teaching 



teclinoiog)' for tlicir own use, but ns the merciful end to n pericxl of 
high anxiety. In such cases, the administrator may count himself 
fortunate if per-formnnce comracting meets only with teacher apathy and 
not with more explicit and disruptive tyixis of obstructionism. 

Realistically, neither of the above stereotypes will characterize a 
particular school system; actual organizational climates and teacher 
attitudes will fall somewhere between these two extremes. The third 
precondition, then, can be stated as a question of degree: "Do the 
system and the teachers in question have sufficiently lew levels of 
success to need a performance contracting program and sufficiently 
high achievement orientations to be able to use one?" If the answer to 
the second part of this question is negative, the school system probably 
has a serious teacher motivation problem v/hich performance contracting 
for student achievement cannot remedy. In this case, performance 
contracting for teacher training may be indicated. 

One other aspect of this precondition demands attention-- the generally 

anti-performance contracting stance of teachers' unions and associations.—'^ 

This resistance is not as uniform as it may seem, for many teachers 

evidently have been willing to go along with performance contracting, at 

2 / 

least ill the experimental stage. Dut where compliance does occur, it 



— ^AFT, UFT, and, informally, the NDA. The University of the State of 
New York, 'I'ho Sta'.e Fduention Department, Performance Contracting in 
Flementaj-y and .Secondary Fducat ion, December, i9'70, p. T3 ' 

-'^Strucker and Hall, The Performance Contracting Concept in Education, p. 






must be described as temporary and ultimately dci?endent on three factors 

a. teacher perceptions of performance contracting's threat to 
tenure, to traditional salary structures, to the prevailing 
professional- paraprofcssional mix, and to the value of 
certification; 

b. union resolutions and local strength; 

c. legal questions about the necessity of developing performance 
contracts through collective bargaining. 

Union opposition may entail so much conflict and such degeneration of 
system- teacher relations that the projected value of performance con- 
tracting becomes marginal. There are, however, several ways in which 
school administrators can seek to preclude the development of unioii 
opposition and even in some cases to overcome it. First, the group of 
educational and training techniques that compose the organizational 
development approach can be used to help teachers "own" the performance 
contracting program and understand it operationally. Secondly, the 
program specifications that prosjiective contractors must meet can be 
adjusted to reflect teacher concerns about automation (through leaching 
machines), erosion of professional status (through use of non-ccriified 
personnel), job competition (from teachers hired by 'the contractor 
outside the system), and other threatening aspects of performance con- 
tracting. Thirdly, before contacting potential contractors, the school 
system can try to sell the concept, not only to teachers and legislators, 



but to tlic community as well. And finally, if the administrator feels 
that his ixirsonncl arc sufficiently knowledgablc and iiiotivated to explore 
new teaching technologies by themselves, he may respond to union opposition 
by considering so-called "in-house" performance contracts on a competitive, 
cost/effcctivencss basis with external proposals. 

A fourth general but veiy important precondition is the nature of the 
immediate school system- -the structures, f)crsonalities, and procedures 
. that directly determine operations in the classroom. This precondition 
can be divided into at least three major questions: 

1. Are current testing procedures sufficiently informative to be used 
as a basis for selecting potential pupils for the contractor's program? 

Have they produced a data base reliable enough not only for grouping students, 
but also foir informing the general design and specific objectives of the 
program ro be contracted? And has the school system developed a 
testing program which inspires enough confidence to permit using it as 
a determinant of contractor payment? Because diagnostic data and resting 
are so crucial to ibe design and operation of a performance contracting 
program, the school system that already has satisfactory testing pro- 
cedures is much better suited to performance contracting, and can 
exercise more effective control over the contractor, than the system 
that lacks such procedures. Moreover, the difficulty, expense, and 
danger of developing such procedures within the context of an operational 
pe]'formance conti'acl cannot l)o overemphasised. Even if the school 



system seeks to nvoicl undue contractor influence in the design of testing 
by arranging with another firm or agency for test development or application 
(usually at significant cost), the intended use of the testing program may,, 
bias its broader validity. The school system that lacks a previously 
developed testing program, in addition, will be unable to verify the 
utility of testing procedures before having to use. them in the very sensitive 
area of payment determination; unable to develop the necessai-}' data base; 
and unable to negotiate from a position of strength when performance 
measures are teing written into the contract. Clearly, the existence 
and perceived validity of a data base and testing procedures are a 
significant precondition, and the school system that lacks such tools is 

well advised to develop them and gain experience with them before it ini- 

. . 1 / 
tiates a peitormance contracting program.” 

2. Is the school system sufficiently flexible to adapt to the contractor's 
program and to take full advantage of it after the turn key phase? This 
is a broad question that includes flexibility of student grouping, of class- 
room scheduling, of teacher utilization, of curriculum composition, and 
of administrative procedures. These elements are important because the 
contractor and his program, as intruders in the regular school system, 
are prone to isolation and ultimate rejection unless they are thoroughly 
integrated from the start; and since the contractor essentially is free to 



— '^Slipulaiions like these also may give school .systems enough expertise 
in evaluation to avoid the notorious "teaciting of the test" that occurred in 
Texarkana. 
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design whatever program he feels will maximize achievement for his 
limited group of students, the onus of integration and adaptation during 
the life of the contract falls on the school system. 

The longer term success of a performance contracting program dci>ends on 
the successful transference of contractor technology. An important 
determinant of this process is school system flexibility during the turn- 
key phase, when the system will be required for the first time to assume 
full responsibility for operating the contracted program. Thus, for the 
sake of ultimate success, as well as for effective, integrated, short- termi 
operation, the administrator must assess his system's flexibility and 
explore its limits before he considers performance contracting. 

3. A related question is whether the school system is sufficiently 
aware of its own educational goals, objectives, and programs to envision 
where a performance contracting program might be productive. Does 
the system engage in the kinds of operational and strategic planning that 
will enable it to set goals and objectives for its performance contracting 
program, to determine appropriate foci for evaluation, and to specify the 
ways in which performance contracting and its goals are supposed to 
contribute to higher level system goals and programs? This last issue 
is crucial because of the familiar danger of sub-optimization, an outcome 
expecially likely with approaches which- -like performance contracting- - 
combine great freedom of design and operation with very powerful incentives 
to achieve. In such cases, the premium placed on performance and the 
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n)mo.st iinljmitci.1 number of nlternativcs tend to focus attention on the 
palpability of ]'osults, and not so much on the relevance of those results 
to system goals. Though all the possibly sub- optimal effects of a con- 
tractor's program cannot be specified and restricted, the major outputs 
can be limited to ihe evidently ge)al- congruent, and evaluation can bo 
designed to discourage certain kinds of sub-optimi?.ation. But the actual 
ability to develop tlese limits and to design related evaluation schemes 

rests on system self-consciousness, a highly variable precondition that 

1 / 

often finds explicit statement through planning activities." 

Finally, the nature of the larger school system and its environment 
are significant preconditions for a performance contracting program. 
Relevant aspects of this precondition include: « 

1. The contracting authority. As pointed out by the Division of 

2 / 

Evaluation of the New York State Education Department,- the legal 
power of a school district to contract for outside services, particularly 
the extensive instructional services of most performance contractors, 
is unclear and varies from state to state. It also varies according to 
the type of funds used, for when federal funds are emploj’ed, ilie contracting 



l/The task of developing such awareness may not be as forbidding as it 
seems, for most school system members rhoi'oughily comprehend system 
goals without having formali/x'd them, d'hc problem then becomes c^nc of 
heiglitcning a pre-existing awareness, frequently through rather simple 
goal-setting pr 'cesscs. 

2 / 

“ Tlic University of the State of New York, The Slate liducaiion Department 
Performance Contracting in Iflcrnentary and Seconda.ry I'kiticalion, pp, 4, 9, 
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agency legnlly is seen as the federal government. But when a federally- 
initiated performance contracting program eventually is turned over to 
the school system, the problenn of contracting authority re- appears, and 
it may require special, enabling legislation. In theory, this problem 
stems from the issues of who is to operate the school sy.stem and of the 
degree to tvhich contracts with third parties constitute abrogations of 
school system responsibility for educational oi>erations: so, performance 
contracts that carefully delimit the contractor’s authority and emphasize 
school system responsibility for program results may do much to nullify 
the legal problem. 

2. Resource allocation procedures. Variations in local resource 
allocation procedures and the relative fiscal influence of the superintendent, 
the school board, and, in some cases, the city council significantly 
affect the feasibility of implementing a performance contracting program. If 
the resource allocation process involves significant political inputs, 
performance contracting easily can become an issue in factional warfare. 

In such cases, the resi:>ect accorded the superintendent and his professional 
staff and the degree to which they are the ultimate judges of discretionary 
expenditures can be decisive. These and other aspects of resource 
allocation- -including the influence on school system policy and finances 
exercised by an informed, articulate electorate- -will do much to affect the 
initiation, continuation, and disruptivencss of a performance contracting 



program. 
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3. Almost equally important are the locally perceived pi oblcms and 
rcspon.siliiliiics of the school system. As opponents of performance 
contracting have noted, it is possible to view this approach as an 
abd'''ation of school system rcsfionsihility, particularly in cases like 
that of the State of Viginia and of Cary, Indiana, where the contracts 
cover, respectively, large numbers of schools and an entire curriculum. 

And the local reputation and histoiy of the school system will largely 
dictate whetler performance contracting is seen as an evasive tactic 
or as an attempted improvement by means of expertise and economies 
of scale in research and development. Similarly, local perceptions of 
the school system's major problems (whether its weaknesses lie in 
particular segments of the curriculum, in personnel quality, in accountability, 
or in some other area) will affect the feasibility of performance contracting 
and even play some part in v/hich aspects of- such a program the school 
system tvill emphasize. The combination of all these local views, both 
w'thin the .school system and among the community, ultimately will affect 
the larger school system's attitude towards performance contracting, 
and determine if it is to be one of parochialism, which is highly suspicious 
of the innovation; of cynical cosmopolitanism, which tream it as simply 
another new approach to student achievement; or one of genuine 
experimcnralism, which is willing to wait for results to corroborate 
either of the preceding stances. 







The long icrni impact of performance contracting on tjcl'.ool systems 
and on society at large is difficult to pj-edict, for the real ability of this 
technique to break through the technological and pedagogic barriers of 
education is still in doubt. It ma> be that performance contracting will 
prove to have been simply another blind alley in the search tor educational 
effectiveness. But that seems unlikely. Whatever its ultimate utility, 
performance contracting will have confirmed the fact that education 
need not remain the preserve of the certified professional, that 
there are alternative methods and alternative personnel who may 
be, in their own ways, as effective as the trained teacher. And it will 
have widened the horizons of school administrators by confronting them 
with organizations whose survival depends on educational efficiency. 

If performance contracting for education persists, we probably nuist 
look for a permanent revolution within the academic profession; for a 
more militant public that has learned it can ask for and receive seemingly 
real achievement gains for its children; and for a profound change in the 
role of the school administrator, who may find himself very quickly trans- 
formed from an inspirational leader into a broker for educational services. 
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AN i:VAM!ATruN OF IMlRKOKMAN^Ci': (:0:rJ10 \CTING FOR H )vl/ 

Pol ly Carpenter 

The HanJ Corporation, Santa Monica, California 

I NTRO] )UCT](>^: 

About two years ago various business firms and school districts 
began discussing a nc\s' type of educational arrangement called a per- 
fomance contract. The firms were offering to instruct students in 
public scliools and have their pay be a function, at least in part, of 
the students’ achievement gains. By achievcMuent gain we mean cognitive 
growth measured by such norm-referenced tests as the Stanford Achieve- 
ment Test or the Metropolitan Achievement Test. At the beginning and 
at the end of a performance contracting program each student takes one 
of these standardized tests. The difference between his scores on the 
two tests is called ”his gain" and determines the contractor’s compen- 
sation, at least in part. 

Performance contracting is a very contioversial concept. It im- 
plies involving profit-oriented firms in public school classroom duty. 
It also implies using test results to measure the effectiveness of in- 
struction and to measure educational output. Both these features have 
produced extensive interest and concern, not only among the educators 
but among the general public as well. The history of performance con- 
tracting has been evident in popular magazines, the press, and televi- 
sion. 

Local educational officials have expressed the need for materials 
which would assist them in deciding about performance contracting 



This paper comprises the notes for a talk on a recent Rand study 
of performance contracting in education, which was performed pursuant 
to Contract No. llhW-OS-70-156 with the Department of Health, Education 
and Welfare. The talk was prepared for the /hiierican Educational Re- 
search Association-American Association of School Administrators Con- 
ference on Performance Contracting to be held in Washington, D.C., 
December 9-10, 1971. 

None of the I'o.marks expressed herein should be interpreted as re- 
flecting the official views, policy, or opinions of The Rand Corpora- 
tion or of the Department of Health, Education and Welfare. 
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program:.; for di.stricts. The U.S* Doparfinenl of Health, Kclucation 

and V.'elfare, in the nuiumer of 1970, decided to sponr.or the preparation 
of a tjuide to meet thir. need. Rand was asked by HKW officials to under- 
take the project pursuant to contract No. IIi'U'“'OS-70-.T!)6 . 

i^ecause any p^uide should he firmly pi'ounded in empirical study of 
actual prc)gra::is, we have been monitoring, in depth, 8 programs in 5 
cities during the past year. We have also been foll.owinp, in less detail 
about 13 other programs or attempts to get programs under way. The re- 
sults of Rand*s work will be presented in three reports available to the 
public. The first report, R-699-HRW, The PevfovtncDice Contracting Concept 
in Education^ was reledsed in Hi\y 19 71. llie second, currently being pre- 
pared for publication, is tentatively entitled Case Studies in Pei^fomnance 
Contracting. The third and final report will be the Performance Contract- 
ing Guide addressed to decisionmakers in local school districts. This 
report is duo. this winter (1971) and will end the project. 

HE\s^ officials were very perceptive forecasters when in 1970 they 
predicted that performance contracting would be a very popular innovation. 
During the 1970-71 school year the Office of Economic Opportunity (OEO) 
sponsored a 20-project, structured demonstration and at least 50 or 60 
other programs were under way, financed from a variety of sources. 

I'/liy the sudden popularity? There are probably two major reasons 
for the great interest in performance contracting. The first is that 
business firms have been asserting that they have the technology and 
skills to close the achievement gap between students from economically 
poor homes and those from more advantaged environments. Those in the 
field of education have been wrestling with America's compensatory edu- 
cation problems for some years with pretty heartbreaking results. They 
are ripe to respond to any promise of help. A second reason for the 
populc\rlty of performance contracting has been a hope that it might be 
an agent for ciiangc. Education is a field within which there has been 
a relatively low rate of innovation. Any mechanism that might overcome 
the barriers to new techniques or teclinology is very attractive. In 
short then, in addition to examining an intriguing new relationship be- 
tv;ecn public and private organisations , the Rand/llEW study deals with 
some very basic- issues of educational policy: the problem of improving 

compensatory education and that of stimulating innovation. 



-3- 



THK R AN])/liia^*jrn 

Many people contr jbut.ecl to our field evaluation of on^’oing perfor- 
mance contracting programs. They represented a variety of disciplines — 
not only education, but mathematics, economics, bu.siness administration, 
and psychology. Our approach was, first, to choose a fev; programs to 
study extensively so that we would really understand what was going on. 

At tlie same time we wanted tlie sample to be diverse enough to permit us 
to generalize tlic conclusions that v:e formed. We also knew that we needed 
a flexible plan because the programs would change as the year progressed. 
Finally, we used a variety of data-gatliering tecliniques ; questionnaires, 
interviews of students, teacliers and prominent people in the community, 
observation in the contracted classrooms and in regular classrooms, 
searches of records in central files and in school files, and, in some 
cases, administration of special tests to program students. 

The cities in which we did our work represented geographically 
varied areas: Gilroy, California, the far west; Texarkana, Arkansas, 

the south central region; Gary, Indiana, and Grand Rapids, Michigan 
(industrial cities), the northeastern region; and Norfolk, Virginia 
(a soutliem city) the eastern seaboard (Fig. 1). Not only were these 
cities geographically varied, their other characteristics were varied 
also (Fig. 2). Population ranged from quite small in Gilroy (around 
11,000) to fairly large in Norfolk (around 300,000). School enrollments 
ran from 5,000 in Gilroy to 55,000 in Norfolk, llie predominant minority 
group in each city was usually black, although in Gilroy the minority 
group was Spanisli surnamed. TTie percent of minority was also quite 
varied; as a matter of fact, the 60 percent "minority" in Gary indicates 
that the blacks are actually the majority in that particular city. 

The programs themselves also were quite diverse. Some of the major 
features of tlie 8 programs studied are displayed on Fig. 3. The contrac- 
tors were Behavioral Researcli Lab in Gary; Westinghouse Learning Corpo- 
ration in Gilroy and in Grand Rapids; Alpha Learning Systems in Grand* 
Rapids (Alpha was part of tlie OLO experiment), CMES (Combined Motiva- 
tion Education System) in Grand Rapids; Learning Research Association 
in Norfolk; and Dorsett Education Systems (during 1969-70) and Educa- 
tional Development Lal> (during 19 70-71) in Texarkana. 



Fig. 1 — Locations of the Five Case Studies 
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CITY NAME 


POPULATION 


SCHOOL 

ENROLLMENT 


MINORITY 


PERCENT 

MINORITY 


GARY 


182,000 


47,000 


BLACK 


60 


GILROY 


11,250 


5,000 


SPANISH 

SURNAME 


34 


GRAND RAPIDS 


200,000 


41,000 


BLACK 


20 


NORFOLK 


300,000 


55,000 


BLACK 


30 


TEXARKANA 


60,000 


14,000 


BLACK 


27 



Fig. 2 — Characteristics of cities in field study 
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CITY NAME 


CON- 

TRACTOR 


SUBJECTS 


STUDENTS 


GRADES 


CON- 

TRACTOR 

SELEC- 

TION 


CON- 

TRACT 

MANAGE- 

MENT 

SUPPORT 


CON- 

TRACT 

EVALUA- 

TION 

OR 

AUDIT 


GARY 




URL 


All 


850 


K-6 


S 


NO 


YES 


GILROY 




V/LC 


READING, 

MATH 


100 


2-4 


s 


NO 


NO 




f 


ALPHA 


READING, 

MATH 


600 


1-3, 

7-9 


C 


YES 


YES 


GRAND 

RAPIDS 




CMES 


READING, 

MATH 


600 


6-9 


s 


NO 


YES 






V/LC 


READING, 

MATH 


400 


1-6 


s 


NO 


NO 


NORFOLK 




LRA 


READING 


250 


4-9 


C 


YES 


YES 


rEXARKANA 




DORSETT 

(1969-70) 

EDL 

(1970-71) 


READING, 

MATH 

READING, 

MATH 


350 

285 


7-12 

7-12 


C 

C 


YES 

YES 


YES 

YES 



Fig. 3 — Features of the eight programs 
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The subject.'; Lau^;lit were* generally reading and niathematics > al- 
though In J’orfcOl: reading was the only subject taught. In Cary, all 
subjects in tlie curriculum were taught, since Cary’s program involved 
tlie entire banneker Klemcntary School. Unlike most experimental and 
demons trat j on programs in education most of these programs involved 
fairly large numbers of students. Tlie Gilroy program wa.s the smallest 
with only 100 students; the Gary program, the largest with 850. Grades 
ranged from kindergarten to the 12th grade. 

Contractors were selected in one of two ways: by sole source se- 

lection (that is, the school district agreed beforehand to give the 
contract to a particular contractor) or by competitive bid (that is, 
several contractors submitted proposals fiom which the school district 
selected one). The programs were evenly divided l)ctween those two 
methods. There were other contractual arrangements. For example, a 
management support group v;as often hired to assist the school district 
in managing the contract, since many school districts felt they lacked 
the management expertise necessary to handle the business relationships 
involved in contracts of this type. Similarly, independent evaluation 
or audit were sometimes contracted on the theory that if an independent 
evaluator were used, there would be no grounds for charges of collusion. 
These kinds of arrangements introduced some difficulties in the programs, 
which will be discussed later. 

EFFECTS OF PERFOT^-IANCE CONTRACTING 

Tlie effects of performance contracting will be discussed in five 
areas: instruction, student learning, program cost, program management, 

and the contractors. Several points will be addressed regarding instruc- 
tion. Did the programs actually change instruction? VJliat kinds of 
changes took place? And what was the extent of these changes? The kinds 
of changes can best be exemplified in a description of the more open 
classrooms in the programs. The typical scene in a regular classroom in 
the inner-city schools had the teacher at the head of the class before 
rows of students seated in desks bolted to the floor. The teacher might 
be leading a discussion, asking questions or reprimanding the students. 
This scene was quite familiar, to most of those conducting the study. 
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‘Jlui scene? in the more open cliissroums of Llic performance contracting 
l^rograms wn.s entirely diffovent. We saw the teacher sitting at: a desk, 
l>os.sibly discussing a pioce of work with a student; in other parts of 
the room some students might be sitting at a table, listening to a tape 
recorder; others would be sitting in study carrels working on workbooks; 
others sitting at tables reading; and still others walking about getting 
materials or putting them away. Trequently there would be a teacher 
aide iielping some of the students with their v'ork or with equipment. 

The contrast with the atmosphere of the regular classroom was quite 
marked. Students in performance-contracting programs seemed to be 
more self -reliatit , more interested in what was going on, more mature 
cabout their leaiTiing than were the students in regular classrooms. 

Another change was that all of the programs emphasized individual- 
ization of instruction in the sense that each student was tested to 
determine his strengths and weaknesses in the program subjects and then 
a course of stuuy was prescribed tailored to remedying his weaknesses 
and to building on his strengths. Thus every student could have a 
slightly different course of study. 

There was an emphasis on teaching the skills of reading and mathe- 
matics for two reasons. First, these are the skills that almost every- 
one agrees are necessary for functioning in our technological world, 
and it is precisely in these skills that the disadvantaged student is 
weak. Second, it is easier for educators to agree on the precise skills 
that everyone should have than it is to agree on what should be the con- 
tent of, say, a social studies curriculum. In effect, reading and math- 
ematics are less controversial subjects than some others that might be 
taught . 

It took time to implement the changes in- instruction even within 
the program, since all of the '‘learning systems" had to be tailored to 
fit tlie many different variables in specific districts. Some of the 
programs did not become fully implemented until almost half of the year 
was gone. At the beginning of the study wc were also concerned that 
the programs would remain "encapsulated" the v;ay so many experimental 
and demonstriition programs do in education — a situation where practi- 
cally nobody outside of the program cares or even knows anything about 
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It. Altlioup.il thifi scH?moci to bt! the caau at the bepinninp of the 1970- 
71 schooJ year, as time went on the proprams set*mccl to be alinor.t selling 
themselves. Teachers became interested in them, and wanted to know more 
about the techniques. ISy the end of the programs, in most cases, the 
ideas had spread to other teachers and schools in the district. In some 
instances, teachers who were initiaJly hostile were won over, especially 
as they began to see that the contractors were not going to be as spec- 
tauularly successful as they had first claimed. 

The key question, however, is. Did perf ornuaice contracts have bet- 
ter than average effects on student learning as measured by standardised 
tests? The measui'e used was the gain score — the difference between a 
student’s score on a pretest and his score on a posttest. Normally we 
would expect that an average student would gain one year in one year of 
instruction measured, say, in the spring of one year and the spring of 
the next. However, in these progran.s this rarely happened, as shown on 
Fig. 4. In Gilroy the reading gain was significantly better than that 
of Other Title I students that year. The math gain was mixed — better 
at one grade and v/orse at another. Since Westinghousc Learning Corpo- 
ration had promised that the Gilroy program would provide better than 
one year’s growth in one year, the program did not live up to the con- 
tractor’s promise. The Westinghouse Learning Corporation program in 
Grand Rapids was similar to that in Gilroy; the gains were about the 
same as those of the Title I students in Grand Rapids in the past. 

The CMES program in Grand Rapids seems to have done quite well. 
However, the gains made are based on results from only a third of the 
student population. Frequently in performance contracting programs 
contract provisions are not included for students who enter late, or 
leave early, or who do not have pretest or posttest scores. Only a 



A standardized test is one that has been given to a large number 
of students throughout the United States so that the test publishers 
can say what score the average 3d, 5th, or 6th grade child, for example, 
will make. A certain student’s score may then be Compared with these 
norms to determine his grade level. 

Referring to Title I of the L..amentary ajid Secondary Isducation 
Act; these are students that are disadvantages in terms of various mea- 
sures such as family income or membcrslilp in a minority group. 




CITY NAME 


CONTRACTOR 


MEAN 

GAliJS 


REMARKS 


GARY 


BRL 


1.7/1. 7 
.7/1.2 


E A D 1 in! G / f /i A T H / 1 5 T GRADE 
REA DING /MATH/2ND-6TH GRADE 


GILROY 


WLC 


.6/. 8 


READING/MATH 


GRAND 

RAPIDS 


ALPHA 


NR 


TEST IDENTIFICATION NOT RELEASED 
BY OEO. THREE TESTS USED 


CMES 


1. 2/1.0 


READING/MATH 


WLC 


.7/.6 


READING/MATH 


NORFOLK 


LRA 


0.1 


READING/5TH GRADE 


0.5 


READING/7TH GRADE 


0.5 


READING/9TH GRADE 


TEXARKANA 


DORSETT 


NR 




EDL 


.5/. 3 


REA DING/ MATH /6TH-12TH GRADE 



Fig. ^ — Mean gains on standf.rdized tests 






IhlrcJ of I lu* ;;t vuk’ii l s in the* CII.S to(*l; llu* full cciurfii* of study 

nnd had both protest and post Lest ‘worcs. Wo do noL know what the (»lher 
fitudcMUs would iiavi* scored. 

The d.'iLa on Llu* Dorsult hditcaLional SysLoins (in Texarkana) prop,ram 
wi.l.] never be avaiJahle because of the alJep.ed teachJnK of test ileiu.s 
in tlic! Doi'seLL program. This i llustraLes an important point. Tn per- 
formance cf»n tract inj;, data may be witlilicld from the public for various 
reasons . 

In Norfolk we have an intcresLinp, situation. At Lie 7th and 9th 
gradd levels the students achieved about what they would have achieved 
in an ordinary program; thus Ixjarning Research Association did not ful- 
fill its guarantee of 1.7 years of growth in 1 year. At the 5th grade 
the gain scores appeared to be almost purely the result of chance. The 
pretest of these students showed many of them to be functionally illit- 
erate, so the teacher chose first to improve their word-attack skills, 
that is, their ability to hear a word and then find it on the printed 
page. For most of the year this objective was pursued, and almost none 
ox the ciiildren progre^sed to reciding comprehension objectives. Dut 
word attack skills were not included on the norm-referenced tests. For 
example, the Metropolitan Achievement Test (MAT) has three sections, one 
of which measures word-attack skills. Since this secticn has to be ad- 
ministered orally, it is expensive to administer and requires special 
training of the test administrator. Thus this section was not used in 
the evaluation. The two sections used measured reading comprehension 
and vocabulary. Therefore it is not too surprising that the gain scores 
on the MAT appeared to be random, as illustrated on Fig. 5. (Gain scores 
on the other two tests used at the 5th grade showed similar distributions.) 
A test of the word-attack skills assigned to the students in the program 
showed that they had mastered these objectives at tie rate of about 80 
to 90 percent. V/e cannot, however, conclude that the students learned 
something from the program^ because they wore not pretested on these 
objectives nor was a control group used. Thus in evaluating programs 
it is necessary that the program and the testing instruments are syn- 
chronized and that base-line data are obtained. 
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Fig. 5 — Gain scores (Mat) 5th grade reading Norfolk 
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The (Jary program appears lo luivo been much more successful tlian 
tlie others, part i.cuJ arly in tlie area of mathematics. Tliere were two 
ol)vious differences between tlie Gary program and tlie otlicrs. Perhaps 
the most important was tljat because the contractor had control of thj 
entire curriculuni, lie was able to concentrate almost exclusively on 
teacliing reading and matli during the first semester. Anotliei important 
diffjience was that Gary parencs were clearly deeply involved in tlicir 
children’s education, in contrast to most of the parents of students 
in other programs. 

Tlie costs of the performance contracting programs can be compared 
to each other and to those of other compensatory programs if the unit 
costs are the saire, for example, if a teacher is assumed to receive 
the same salary whether he or she is teaching in Gary or Norfolk. On 
this basis, the costs that would be incurred yearly for instruction for 
reading or mathematics (there was no significant difference in cost be- 
tween reading and mathematics) of the performance contracting programs 
would be similar to or sometimes 25 percent less than those of the 
regular Title I programs. * This is because the contractors substituted 
materials, aides, and equipment for teachers. For example, in the Nor- 
folk performance contracting program the teacheir handled 25 students in 
the classroom, five periods a day. In a comparable remedial- reading 
program under Title I in Norfolk, the teacher handled 10 students live 
to six periods a day. Moreover, when we asked the Norfolk teachers in 
the performance contracting program whether they felt the class size 
was too large, they said they thought it was too small. They could 
have handled more students if the room had been larger. 

Compensatory programs, including performance-contracting programs, 
should be and arc more expensive thaii regular programs by 60 to 70 per- 
cent. After all, these programs are trying to make up for deficiencies 
not only in the student’s education but in his home environment as well. 

Performance contracting had an effect*, on program management. We 
know that not only scliool systems but almost all institutions are short 
of the management talent that is needed to effectively implement change. 
Fortunately, in each of the programs there was someone willing and able 
to take over the duties of effectively implementing the program; in 

r; 
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SOUK! Giber pro^;rams lack el’ Icaili*r.slnp rcsulleci in inef foci j.vc imple- 
nientalion. pAicauso tlie pro^prains were clevelo|)!nent al in niiture, flexi- 
bility of manaj;ement v;ar» neecled» There were tiincs v;hen tlic contractual 
arranp^ements actually hindered cliangc.s tliat sliould have been made to 
make the prof;ram more effective. 

Altliongl) we would expect qualified teachers to be involved in pro- 
grams of this type, some of the . contractors felt tliat they had programs 
tliat were "teacher-proof" and even went so far as to try to show that 
no matter what kinds of teachers were assigned the program would work. 
By the end of the year, however, these contractors had realized the 
importance of mailing use of the teadiers' knowledge of the student pop- 
ulation as a valuable resource in implementing their program. 

A number of people in the school district had to assume new roles. 
In order for the contractors to make good their claims that their sys- 
tems could teach they had to actually become involved in the instruc- 
tional process. School administrators had to be sure that they lived 
up to the terms of the contract; that enough students were available 
at the right time, that the rooms were ready, and t'nac tests were ad- 
ministered properly. And teachers had to become managers of instruc- 
tion rather than presenters of information. 

What advantages did the contractors receive from their programs? 

We are not sure that they made any money. None of them fulfilled his 
performance guarantee except BRL and CMES, but because we do not know 
how much the 1970-71 programs actually cost the contractors, we cannot 
say whether they gained or lost. Although monetary gain has been pub- 
licized as one of the reasons contractors are interested in performance 
contracting, we think they are much more interested in obtaining new 
markets and follow-on contracts. Most of the contractors have small 
organizations that have had difficulty breaking into the educational 
field or providing competition to major textbook manufacturers. The 
performance contracting mechanism gave them a way in because of the 
performance guarantee. Follow-on contracts have resulted from almost 
every program that we have monitored. Westinghouse Learning Corpora- 
tion in Gilroy Is the only one wiLliout a follow-on contract because 
Westinghouse lias decided to close down its Learning Division. However, 
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even in Gilroy teachers want to contir*ue the Wer.tinghousc approach in 
their own learninj; ccniters. The demise of WestJn^;house in Grand Rapid5> 
was folJowed by tl;c formation of a company called Learning Unlimited, 
which is going to continue the Westinghouse type of operation. 

AnVANTAGLS AND DISADVANTAGES 

Performance contracting has some disadvanta.ges and problems. The 
disadvantages are complexity and narrowness. The technique is inherently 
complex because tl*e. contractual arrangement introduces new problems in 
assignment of authority and responsibility and the management of the 
new program becomes more involved. The technique is currently narrow 
because the performance contruct must focus c*‘ skill are^.:, ^ since the 
definition of objectives and the construction of tests are difficult 
In more controversial subjects sucli as art or social studies . 

There was also an exacerbation of old problems. Management prob- 
lems may be accentuated; legal difficulties may arise from contractual 
arrangements, which may sometimes violate the education code or agree- 
ments with textbook manufacturers or teacher ujiions; problems related 
to tests and test administration will be more obvious because measure- 
ment is so crucial to performance contracting; finally, the issue of 
teacher status has caused difficulty with teachers and teacher unions. 

Performance contracting offers two important advantages for teach- 
ers and students. First, it really does facilitate radical change. 
Outsiders to the educational system are freer to implement change than 
are those wlio are part of the system. Second, performance contracting 
has caused an increased emphasis on accountability: The schools now 

feel they are accountable for student learning, rather than for inputs, 
such as the teacher/student ratio or teacher's salaries. The empliasis 
on student learning is a very healthy thing. It has forced the con- 
tractors to look very carefully at their systems and to understand what 
their claims mean in terms of student learning. School administrators 
have become more aware of the problems of evaluation and testing. 

Teachers have become more aware of the necessity for their students to 
measure up to other students or to their ov;n past performance; and the 
students themselves may become accountable in the future. In fact, 
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ruinor has it that some students in n small town in Texas complained 
that, since they were the ones ultimately responsible for their own 
learning, a performance coi\tract let to a nearby educational firm should 
have been given to them directly. 

In concliJsSion , we think performance contracting will continue for 
at least a few years, in spite of some of the poor showings on student 
achievement. Certainly the follow-on contracts suggest this, even 
though some of them do not have performance guarantees, and new con- 
tractors are entering the field. Performance contracting is also a 
helpful change agent and does provide the emphasis on accouiitability 
that is currently in vogue, Wliether the technique will have any last- 
ing benefit we cannot say. First, considerable work will be needed to 
develop measuring instruments before performance contracting can be of 
wider use. 
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Some Problems in Assenslnp, Pciucation.'tl Pcrformcuico 

Robert I.. Kbel 
Miclii[;an State University 

1* Rackjjrouncl and overview 

The r:ssessi:?on t of perCornnnee is as old as liuman history. The last 
verse of the first chapter of tlie bool; of Genesis says, ’'And God saw every- 
thiny tliat he had made, and behold it was very Rood.” Since tlien most of 
us v.’ho v;ere made in His image have had many occasions to follow his good 
example, though often without the same satisfying conclusion. 

Assessing tJ^o educational attainments of students, directly, and thus 
of their teachers, indirectly, also has a long ’history . It began at least 
A, 000 years ago, for there are good records of an elaborate system of achieve- 
ment cxar'.inaticn*^ j which provided entry to the civil service, in the Shun 
dynasty of ancient China* It continued In the middle ages, llie Jesuits, for 
e>:a?nple, prepared a detailed and technically sophisticated set of rules for 
the conduct of v;rittcn examinations which, if followed carefully, would 
iTiprovc n;any of the examinations we give today. 

Assessment played an important role in the early development of public 
education in this country, at die initiative of Horace Mann and others. It 
beg.in to become a science, or at least a technology, in die early year's of 
this century, in the hundj of men like Cattell, Tlionidike, McCall and Wood. 

Nov, v.’itli die proliferation of wide scale testing programs, with tlie perfec- 
tion of electronic test processing equipment, and with public pressure for 
accountability in the educational enterprise, assessment continues to play a 
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lending, roJ.p in t;lio [/locejiG of education. Surely it Js an csjieritlnl pare of 
any '■.ystem of per foi'iaance contracting. 

The process of assessing performance in education, however, is much more 
complicated t.han it appears to be to many of our fellow citizens. As they 
see it, all one lias to do is to give a test. Each student either passes or 
fails it. The proportion that passes indicates the effectiveness of the 
educational program. Where the test comes from, what it really measures, how 
good it is, what tlie scores mean, how the passing score is determined; those 
and many other hard questions that trouble test specialists a great deal 
trouble the general public hardly at all. They trust our technical compe- 
tence to do tlie job that needs doing far more than it ought to be trusted. 

For even am.ong us educational specialists there are confusions, uncertainties 
and fharp differences of opinion. Tlicre are, in short, some difficult 
problems in assessing performance adequately. 

lliis paper is addressed to some of the more important of those problems. 
It is organized around seven questions which pose those problems: 

What performance should be assessed? 

How should performance be assessed? 

Who should assess cducatloniil performance? 

How can tlie necessary tests be obtained? 

How can tti.To.hing to the test be prevented? 

How should the test scores be interpreted? 

What level of performance sliould be considered satisfactory? 

Clearly these problems of assessment are not- unique to performance contract- 
ing, They arise whenever a performance uliat can be learned is to be assessed. 
But some of tlicm have unique facets, or present unusual difficulties, when 
fulfillment or non-fulfillment of a performance contract is at stake. These 
special problems will be taken note of as the more general problems are being 
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2. What perfonnaiu:o should ho a.sno.sscd? 

A cljroct cinswor to Mils cjuostloii is, *^rhc por formau uo you’ve bct'n trying 
to teach." hut this answer is not as simple or as ov.sly correct as it 
seems at first glance. Some would dory that sch. ols should l>c concerned pri- 
marily with teacliing pupils to poriOrm. They v-’Quld argue tliat schools should 
aim to develop the pupil.’s l^nor ledge , uv;\ ^rs tending , attitudes, interests and 
ideals; that schools should Uq concerr.ed with cognitive capabilities and 
affective dispositions ^uther Uinn with performances, lliey would agree that 
capabilities anl dis -osltions r.jr. only be assessed by observing perf onnances , 
but w’ould insist Lhat the performances themselves arc not tlio goals of 
achieveraent , ^ ut only the indicators of it. A teaclier v;lio is concerned with 
the pupil’s cognltivj capabilities avid aff^^tivc dispositions will teach 
quite differently, they claim, t’ lu one v/hnse attention is focused solely 
on the pupil’s performances, /uid, if pcrfcrr/.anccs arc not goals but only 
indicator , we sl^ovild choose the ones to use in assessment on the basis of 
their effectiveness as indicators. Clearly we cannot choose them in tcmis 
of tlie amount of effort we made to develop them. 

but, if we reject porformauco goals, anotlicr question arises. What 
sliould be the relative emphasis placed on affective dispositions as opposed 
to cognitive capabilities. Here is anoth.er issue that divides professional 
educators. To some, how the pupil fecl5> ; his happiness, his interest, his 
self-concept, his yearnings arc wliat sliould concern teachers the most. To 
others the pupil’s cognitive resources ^md capabilities arc the main concern. 
Both would agree that cognition and affect interact, and that no school ought 
to concentrate solely on one and ignore the other. But, tJiey disagree on 
which should receive primary emphasis. 
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In tryltif; to rcnolvc this Imsug It may be helpful to observe two thlnp.n. 
'fliG first i.'i that ilu/ Instructional pio^.rnms of almost all scliools are aimed 
directly at the cultivation of co[;nltive competence in such thlnp.s as reading, 
uia^fiomatlcs , written cxpre.ssion, science, history, etc. llio pupil's affec- 
tive dispositions; Iiis interests, attitudes, self concept, ideals, etc. 
constitute conciltions tliat facilitate or inliibit cognitive achievement, lliey 
may be enlianced bv success or impaired by failure. The rules of discipline 
and order adopted anc enforced in the school, the models of excellence and 
humanity provided by the teachers, these and other conditions in the scliool 
have pox’erful affective influences on pupils. But they are byproducts, not 
the main products of the instructional effort. It is almost impossible to 
find any school that has planned and successfully operated an instructional 
program aimed primarily at the attainment of affective goals. 

4 

Tlie second thing Co observe and remember is that if a school should set 
out deliberately to develop certain affective dispositions, It has only two 
effective means to use. One is the cultivation of cognitive competence, of 
knowledge of good and evil, of consequences likely to follow certain acts. 

The other is conditioning, the use of rewards and punishments to habituate 
students to seek this and avoid chat. Conditioning is a powerful tool. 

Hitler and his colleagues used it with great success in Germany, particularly 
in the 111 tier Youth gi cups . But, clearly it is also a dangerous tool. The 
cultivation of cognitive competence is far better suited to the education of 
intelligent beings for life in a free society. Hence, we conclude that the 
emphasis most schools do place on the cultivation of cognitive competence is 
not misplaced. 
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V.'liat ia cc^’.iil 1 1 v»: ciniipo tcncc? 'fwo dl.stlnctly cKffcrf’iit an.^wcr.'! linva 
bciMi ('IvcMi. Ono is that it requires acquisition of knowledge. The other Is 
that It requiroH developnout of intellectual skills. Here is another is.sue 
on which eclucational .spc'ciali.'i ts are divided. 

To avoid confusion or suporfi ciall ty on this issue it is necessary to 

be quite clear on the meanings attadied to the terms kncrwledpe and 1 n telle c - 

l\\i \ I s kills . Knowledge, as the terra is used liere, is not synonymous witli inforiiuitio 
It is more nearly synonymous v/ith 

under s tending . Knowledge is built out of Information by thinking. It is an 
integrated strueture of relationships among concepts and propositions. A 
teacher can give his students information. He cannot give them knowledge. 

A student must cam the right to say, "I kncA. by his own thoughtful efforts 
to understand. 

VIhatever a person experiences directly in living or vicariously by 
leading or listening can become part of his kno'wleoge. It. will become part 
of his kncn^'ledge if he succeeds , in integrating that experience into the 
structure of his knowledge, so that it makes sense, is likely to be remembered, 
and will be available for use when needed. Knowledge is essentially a 
private possession. Infoimiation can be made public. Knowledge cannot. Hence, 
it would be more appropriate to speak of a modern day inlonnation "e.xploslon" 
than of a knowledge explosion. 

The tenn intcl lectual ski 11s has also been used witli a variety of mean- 
ings. Turther, those who use it often do not say, precisely and clearly, 
what they mean by it. Most of them seem not to miean skill in specific 
operations, such as spelling .a word, adding two fractions, diagraming a 
sentence or balancing a chemical equation. Tlicy are likely to conceive of 
Intellectual skilLs in much broader terms, such ns observing, classifying, 





mOti.su r i , ccJininuti I call n{; , prodl c llnp, » t nf or o):pc!i*i monl liij» , f onuulallnp, 

hypoLlu'So.s , and intorp ruling data. 

It scoins clear that tlucso broader Intclljctual skills cannot be developed 
or used very elfectJvely ajKirt from substantial bodies ol relevant knowlodp,e. 

To be. skillful in fonmilat i.ni; hypotheses about the cause of a patient's persLs- 
tent lioadaches one needs to know a considerahlct amount of nourolop,y, anatomy 
and physiology, as much as possible about tlie known cHsorders that cause 
headaches, and a great deal about the history and habits of the person who is 
suffering them. That is, to show a particular intellectual skill a person 
must possess the. relevant know’ledge, (Note ».elL at this point that a person 
cannot look up the knowledge he needs, for kn<^% ledge, in the sense of the 
term we are using, cannot be looked up. Only infonaation can be looked up, 
Kncrwledge has to be built by the kner.^er, himself.) And, if lie does possess 
the relevant knowledge, what else does he need '.o shc/w the desired skiliV 

intellectual skill that goes beyond kncrvledge can be developed in 
specific operations like spelling a word or adding fractions. But, the more 
general (and variable from instance to instance) the operation becomes, the 
less likely it is that a person's intellectual skill ’Jill go far beyond his 
knowledge. 

loose who advocate the development of Intellectual skills as the princi- 
pal cognitive aim of education often express the livllef (or hope) tliat tliese 
skills will be broadly trunsfe.rrablc from one area of subject mutter to 
anotlier. But, if tlie subjects are quite different, the transfer is likely 
to be quite limited. Wlio would hire a man well trained in the measurement of 
personal cliaracterlstics for the Job of inea.suring stellar distances and compo- 
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Our coiic ] u'jic'n .it tlilr? point in that. sc!iool;i witicli omplinolr.i: cojjnltivn 
ncliiovcMnonCc , wlilcli la the: vn:»t ii'.njority of Llictm, should coiiLlnue to cultl- 
vnCo. h.novlc'dgo rnthor tlion rionc’ othor tlwm Vuio'^lcdp.i: called Intellectual 
skill. Ihey should continue to do so at least until intellectual skills have 
been more clearly defined, and their contributions to effective performance 
v;hlcl> are different from and Independent of knr/vlcdgc have been positively 
Identified. 

3. How should perfonnance be assessed? 

Two small questions and one large one arise when we begin to consider, 
how a learner's educational performance should be assessed. Let us take up 
the small ones first. 

Should performance be assessed by observing the learner's behavior in 
natural everyday situations or in somewhat artificial clrcuras tance.s of a 
fomal test? If v.’c had enough skilled observers so that individual learners 
could be followed and observed closely as they worked their way through the 
everyday situations they encounter, if we could get funds to pay for such 
extensive and expensive observation and if the learner would continue to 
behave naturally in the presence of ".he ob.server, then the first alternative 
might be worth consideration. But, natural behavior is troublcscxne and 
costly to observe systematically, and most difficult to assess preci.sely. 
Hence, assessment via testing is almost always preferred because of it.s 
convenience and uniformity and because of the reliability of the assessmt*nts 
It yields. V.’hen educational achievements of the kind schools spend most of 
their time developing are to be as.se.ssed, die error introduced by the artifi- 
clalit)' of the Le.stlng situation tends to be small. 
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If tests arc to he ur;c(J> should they be C5.say or objcctlvu? If ono 
wishes to assess tlu* quality of a stiui('.nt^s prose stylo, or of his handwrit- 
ing, the use of an essay tost is 1 ndispc'nr.able . but, if one wishes to asrjess 
his structure of kncrw]ec!^;e lii an area, the objective test can do the Job just 
as adequately, or more so, at the cost of much less time, labor and money. 
Objective test.s need not be, and seldom are, in fact, limited to testing 
superficial recall of factual information, l^ittle blind guessing is done on 
appropriate objective tests by well*“motivated examinees. Tlie folklore that 
disdains objective testing dies hard, but it is clear that such tests provide 
tlie only practical means of obtaining valid assessments of performance on any 
large scale. Education in Ariierica is a large-scale operation. 

Con-sider new the larger question that relates to the h ov; of assessment, 
ould the tests bj designed to yield nonn-ref erenced or criterion ref*^^ced 



m^i5uref- ? A criterion-referenced test is intended to indicate how much of 

t ' 

some desired achievement a given student has managed to acquire. A norm- 
referenced test is intended to indicate how the aiaount one student has 
acquired compares with the ai^.ounts other students have acquired. 

It may be useful to note, at tnis point, that the score on any achieve- 
ment test can have both criterion referenced and norm referenced meaning. 
Tliat is, it can indicate wdiat proportion of a set of ^sks of a' givan kind 
a given pupil can successfully complete. If scores o^r^ne^Iiupils"^on the 

'"W 

same test are available, the same score can be used to dotcrmlhp whereat 




stands in the group; high, middle or low. Both indications are useful 
tl\e assessor. Tl\c issue before us, then, is not whether one kind of mecaning 
should be attended to exclusively, and the other disregarded. It is whether 
we should build tests so as to bo most effective in providing criterion- 
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ro3aLO(l or norn- rol oLod io<:rinl . Makiiig a tost do one of the* two Johi, Ijcttor 
Ir. Ilkcly> at joast In ihooiy, to make it do tiu! other somciwliat lesa well. 

7 \\o. key tu the tlieoretlcal difference between tlio tv/o kinds of tcisLn Is 
in the Btandrird-^ \n>cd to select the items. Vor a criterion-related tctsl each 
item sluxild be written 550 that a correct answer li.dlcatea attainment of r*omc 
important eleiatmt of acdi ievemcin t: in the r;ubjoct of the test, lliat the item 
may be ansv;ered correctly by most examinees, and, hence, diccrlmate pooi'ly 
(by the usual rieasnires of item discri iiilnatioi ) is of lettlo concern. To the 
constructor of a cri torlon-referenced tost, the relevance of the item to the 
goals of aclneveinent is more important than the information it provides on 
differences in achievement among students. . T.is is the theory, at leacL, 

The constructor of a norm-referenced test is also concerned wltli the 
relevance of the question to the goals of Instruction. But, in addition, he 
is likely to pa.y attention to herj nnic’n information the item provides about 
differences rumong pupils in achievement. Given l\:o items of nearly equal 
relevance to the goals of instruction, he will choose the one tluat c*^scrirnin- 
ates better betn/een stude ts of high and low achievement, Tl)e constructor of • 
tlie criterion referenced test would be expected to discount evidence of dis- 
criminatory pov;er and clioosc the itcui which neemed the more relevant to the 
goals of instruction. 

On closer examination the theoretical differences between norm-referenced 
and criterion voferouced tests turn out to be loss than might appear at first. 
Relevance, an v.»e have noted, Is iiupo^'Canu in both tests. So is discrimination, 
dosjjlte disclaimers from proponents of criterion referenced testing. For the 
items in a criterion refc^rcnced tc\s t must discriminate those who have attained 
a particuLir goal from those who have not. Both kinds of tests require good 
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toot J for v;lil(;h citii^wcrs (1) c;iti lu; clffoiKicd nc correct, (2) are not 

conecaJed fro:n capaMu students by a;;;b l;'ul Ly or inaccuracy of expression and 
(3) are not £;ivcn away to the incapable by uni nten tional clues. A good cri- 
terion referenced test might be remarkably similar to a good norm-referenced 
test on the same subjects. VJitli labels renoved even a competent judge might 
find it difficult to tell vliich \;.is which just by looking at the items. 

Preference lor criterion referenced tests is often based on tvo rather 
unrealistic assumptions or expectations. One i.s that in a well-taught course 
most of tlie students will attain most of the objectives of instruction. If 
they do, even a good test item would she.'/ little discrimination among 
.students, for no important differences among them exist to be discriminated. 
But, individual differences among students in background, motivation and 
achievenent cannot be aboliched by Procrustean techniques of ‘’mastery learn- 
ing,” though th*ey may be obscured by repeated tcn’.ting until a, catistactory 
performance i« attained. Even a homogeneous group of students > when faced 
v;itli a challenging course of instruction is likely to slicr.^ substantial 
individual differences in achievement. Good items testing that achievement 
are likely to disci ininate bet^N^ecn individuals of high and oI low achlcvencnt. 

A second unrealistic assumption is that by basing each item in tlie test 
on a separate important objective of instruction, the constructor of a 
criterion referenced to.Kc can obtain scores that more accurately Indicate 
hov; much a student has actually learned of what was set out to be learned. 

The trouble with this notion is that different objectives are not separate, 
discrete entities, independent of each otlier. Ihey are all interdependent 
parts of a complex whole. They tend to r4jsist clear definition a« distinct 
car.ponents, for they overlap and Interact. Success on a particular discrete 
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t(.'s t iLcin alii;o.*:t m;vc»r Indlcato:^ that one particular diccrotc olijectivo, aiul 
only that object Ivo, l\a.‘i been attained. 

"/jiiount to bo learned" .•eid "amount a j;lven .'itudont lias learned" arc hotli 
vary difficult to (luantJ fy precir.ely. It is usually safe to infer that tlie 
student who answers more questions right on tlie test lias learned more about 
the subject matter of tlie test tlian the student who answerr. fewer questions 
correctly. But, except in a f ev; spoclal case.s like spelling or the basic 
addition facts, it is inappropriate to use a tost score, even a criterion- 
referenced test score, as a basis for estimating the fraction a student h.is 
learned of what it was hoped he migh t. learn . To make matters worse, the 
hoped-for amount of learning is usually based, on the private subjective 
judgment of a particular teaciicr or test constructor. Seldom if ever is it 
based on generally iiccepted, clearly defined standard of acliievement . 

Frccii these consideration.', we conclude tl at criterlcii-rcferenced testing 
offers no great prcnlse for substantial improveineiit.s in the asses.sment of 
educational performance. Test scores need to indicate, and most test scores 
do indicate, both ho'.v' much a student has acliieved relative to the subject 
matter and how much he lias achieved relative to his classmates or peers. 

Where v/c can make those indications more precise or more valid we should do 
so. But, we should not expect criterion-referenced testing to express the 
cccnplcexi ty of educ.ational achievement v;ith high precision on a simple 
numerical scale. 

^1. l'.^lo should a.ssess educational performance? 

In the educ.itlonal systems of the Onited States, student performance has 
u.sually been assessed by the pcr.son who tried to develop tlie performance 
capability, that Is by the teacher. Efforts of educational leaders, govern- 
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mfilUal aj'.ancic'S or the* put)]Jc to Inlrocluco cxK'rnal as.^ossinen t?; of parfor- 
inance have becMi siron;;ly resisted l>y school adni nls trnlors and classroo^n 
teachers. , 'Hiey have jiwlifled llu?i r apposition on these grounds aiaonjj 
others : 

External evaluations of achievement v/ill lead inoviiably to 
external control of the curriculum. 

Externally produced tests arc not ccr.:pletcly relevant to the 
local i n.s truct J onal objeclives, and hence lead to invalid cvalua-^ 
tions, 

Tciicliers, tlireatcned by possibly unfavorable assessments, 
will abandon effective pupil-oriented teach.ing and instead teach 
to the tests via rote learning and drill/ 

External tests place too much emphasis on easily measured 
but lerc essential outcomes, i.e., rneall of factual information, 
to the neglect of more important but less tangible outcomes in 
the area of personality, adjustment, character and values, 
lliere is some truth, but much more error in these allegations. External 
assessr'.ents can be harmful to the process of education, but they need not be. 

Properly eiaployod, they can do much more good than harm. Teachers and scliool 

# 

administrators need to face the fact that external evaluations of their 
cf f ectivenesji is inevitable. Tlioy cannot escape accountability to students, 
parents and taxpayers. T\\e question of external evaluation is not whether 
but \\m and hw uuich and by whan. 

Both the process and the product of education need to be evaluated. The 
process can be evaluated informally by students, parents, supervisors and 
teachers, and foraally by specifically designated teams of observers. TI\o 
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pr<uluct can alrio bv‘. evaluated informally on the ba.sis of incidental ('’/!;ei'va- 
tlon, or forniail.y on tlic basis of tO'»t pcrfonnance. It is (he use of test.s 
in af>yesf.lny tl^o product of oduention that is of most concern to us here. 

Who should make tlie tost? Even for an external evaluation the test 
fhould be planned by a team Including both local, teachers and outside evalu- 
ation specialists, llie tost should tl\en be developed externally and adminis- 
tered to pupils by soineono other than their rcp^ulai' teacher. Relevance, 
expertness and Impartiality are the !;eys to effective external assessment 
via tests of student performance, nic teste used must be relevant to tlic 
goals of instruction, 'flie tests should be developed or selected by experts 
in tlie measurement of educational achievement. The tests should be adminis- 
tered and scored, and the results interpreted, by persons who have no 
responsibility for developing the performance which is being assessed. 

Important as it is, external assessment provides only part of the ^ ^ 
information needed for a comprehensive, formal assessment of performance. 

Tlie other main part should come frexm the teachers themselves, using tests 
they have designed and built, or selected from available standardised 
instruments. A balanced program of formal assessment will include both 
internal and external tests. 

What has been said in the preceding paragraphs about assessments of 
tlie process and product of education in conventional instruction applies 
with even greater force in performance contracted instruction. Relevance, 
expertness and impartiality are again the keys to success, 'fliey also present 
difficult problems. Because performance contractors seldom have tenure they 




are likely to be even more concerned than tcadicrs typically are to have 
their methods and results assessed .fairly and adequately. Because of the 



money nful .•ipcclal ufforL involved, tin* sclioo.! iuifhorit llhewito have a 
larjjo aCalu* In the qii.-iliLy of tlic asyossjuc-nt • I'aiJurc to do a coiniietont, 
ccimpreliCMi:; i VO J<)b of ai.'Jics.'iim.'nt could dlacrf'di c the wliolc* cntcrpvlse of 
pc rf orinancG contractin;’ . Doinp, a competent job is not easy, bnt it is 
clearly possible. 

5. How can the nccoa.sar)’ tests bo obtained? 

Tlic two c*bvi(ni!: altc-rnative answers to tills question arc (1) to find, 
select, and buy a satinfactoi'N* ready-made standardized test, or (2) to 
develop or employ sexaeone else to develop a satisfactory* custom-made te.it, 
built to particular specifications. Tliere are pros and cons to each alter- 
native. 

Hie easy availability, relatively low cost and good repute of a widely 
u.sed standardized tost are persuasive arguments for the first alternative. 
Furtb.cr'.v.cre , pul* li slice! standardized tests are usually provided with noiTi ; of 
achievement based ein t!ie performances of large numbers of pupils from more or 
less well defined groups. Such noms are a considerable help in score inter- 
pretation. However, such tests are not available in all, or even most, of 
the special field.s of learning pursued in the schools. Even if a test with 
the proper title is available, it may turn out that the tasks it presents are 
not highly relevant to the objectives of instruction in a particular course. 
Worst of all, the easy access toacliers or performance contractors have to 
published standardized tests opens tlie door to specific coaching of pupils 
to do well on that particul.ir test. Tliese problems are not insoluable. Some 
teachers and perfonaance contractors have made effective use of standardized 
tests as external a.ssesstnentK of performance. In many situations, however, 
the di .^advantages of using standardized te.sts will outweigh the advantages. 
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'/.le iT llc! nia l lvf‘ to u>ln^; a Man J.ircll zed test is to build o'- '.-..ivo l»utJC, 
n tent deslyiu'd especially V.o coiurllnite d ,;.u lo a part J. culni- assessinfMit 
prc’i' Viilji appro, icli is likely to be more costly, but Co result in a test 

of grcaCiir re],.'vanco. Tlu: technical quality' of the test may bo soniowliat 
Icr^'er. Ilcw tiiucli lover depends, in pvart, on hov much money Che user is v.'illln^; 
to St :r.d for its development. Finally, the possibility of linrmful teaching 
to t!. ; tent can bo min imi ?.efi by tin's appro, ach. 

The devo.1 C'V'mcnt of a custom-made te.st for assessment of performance 
should begin in a conference betvecn teachers (or contractors) and test 
developers in which the objectives of instruction are stated clearly and 
comprehensively, a table of specifications developed, and acceptable proto- 
type items agreed upon. Having equipped themselves with the textbooks, 
manu.-.ls, learning programs, and oclier materials used in instruction, the test 
de.vc 1 •: po rs then proceed to the writing, review’, editing and assembly of items 
into a test. If a remote, protected trj’out .ind analysis of the item perfor- 
mance can be arranp,od, so mudi the better, 'if not, the scores obtained from 
operational use of the test ought to be analyzed to provide part of the basis 
for judging its adequacy. 

good CDS to.m-made test can be a ver>’ costly item. Ten thousand dollars 
to $1'. ,000 is not an absurd upper limit. But, in proper hands, a reasonable 
facsirile of a test of highest quality can be produced for about one-tenth of 
those 'Upper limits. It is not necesscU^ that each test be designed and built 
with ccmpletc originality. If an existing test approximates the requirements 
of th.e new tost, it can bo used as at least a partial template or model for 
the nc-w’ test. 
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6. llov.' c;m foch i nf’, to tlu' lusL br prrvunt.cd? 

If Llic. rf ft'cL l.voiH'.’i.v of J nr. tnict I oo is to he • a?>r,osr,«jd ou tlui basis of 
sLud'-fiit porfotmancc! on n tost, tlio tPi:;p ta Li on is stronp, for tlio instnicfor 
to propare ntmlonts to liamlJo tlic* .spoclfic questions tliat will bo 1110111110.(1 
in Llio tc.st, Thl.s is often called "toacliing' to tlio test." Obviously, it 
is not an educa 1 1 on-j 1 ly beneficial procedure. Obviously, it tends to spoil 
the test as a iie-a^iire of general achleveiaont in the subject of the te.st. 

But, if the test is a readily available, publis'icd test, an insecure or 
sliort-sighted teacher is quite lihely to undertake to teach to the test. 

Even a Lest specially constructed for a particular assessment task is, in 
effect, a "published" test once it lias been used. If teachers care enough, 
tliey can find out a great deal about the particular questions included in it. 
So, it too may invite "teaching to the test." 

Several things can be done to lessen the likelihood of teaching to the 
test- Teachers can be warned to avoid it and informed that their supervisors 
will be alert to notice it if it occurs. Pupils can be advised of its 
undesirability and a.sked to report its occurrance. Of course, the surest 
but moat expensive way to forestall it is to prepare a new test for eacli new 
assesi'.nent effort. 

Before leaving this subject, let us make an important distinction 
belvccn "teaching to the test," i.e., attempting to fix in pupils minds the 
answiTS to particular test questions, and teaching material covered by the 
test, i.e., attempting, to give pupils the capability of answering que.stlons 
like those in the te.st on topics covered by the test. Tlie first is 
thoroughly reprehensible. The second reflects purposeful teaching. Just 
as thovo. is no warrant for giving away the answers to particular questions 



no Lliore Is no \/;n;r,'int for Lor.tJnj; pupil porforir.iince on task?! tlioy were 
never toupht to i)Crfon!i. A teacher oi' contractor ^^;ho.se v/ork is to be n.'istissod 
is entitled to l;nw what his pupU;i v/ill be expected to do. This calls for 
close coojKM'at ion and clear co:!iinuni cation between te.achern on the one hand and 
tcjjt contructors on the other. An a.ssesament teat must be thoroughly relevant 
to the instruction it is intended to assess. Since a test can never elicit 
more tlian a ;;an;plc of performances, much more will usually be taught tlinn can 
be tes..ed. Hut, the test should never go beyond v,’hnt has been taught. 

7. How should the test scores be interpreted? 

Tlie score on an objective test usually begins as th.e number of questions 

answered correctly, from this one can infer content meaning, e.g., what 

proportion of a hyi)Othetlcal population of similar questions th student 

could also answer correctly, or roughly what fraction ho knov;s of what he 

night knew ab.-^ut a particular field of ];ncr..’J.cdgs . From the number of correct 

answers given by eacii student \.’bo takes the test v/e can infer normative 

meaning, i.e., v/e can determine v/here eacli student ranks in relation to his 

classmates. If v/i* have determined the reliability of the scores, v/e can 

estimate hov; accurate the scores are, that is, how likely it is that a 

student's actual test acore differs fro:n his hypothetical true score by 1, 

2, 3, /<, or 5 score units. Other interpretations, v/hich are 

scmeti.mes suggested, raise questions. 

These are all useful interpretations. . For example, should a student’s 
score after l\is troction be interpreted by itself or in relation to his score 
on a similar test before lv<-; vructlon? ITiat ic , sliould we measure gains as 
vz-cll ao , or in place of, status? 

At first glance tlie answ'er is obvious. It is the improvement brouglit 
about by Instruction that should interest us, and delight or dismay us ns 
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tlio c.TSfi i.iny he. Unf ortim.ucJiy , sicoros .iro notori ou'’.’ J.y imro lliihle . If 

Llio corrclat-ion he t-vu'cn [U’otest; .'md po.s t- tes f in moderaU'ly hlpji, ;ik It ofLen 
is, most of tlie true ncorc- vnrijinco disappears from the difference .scores. 
What is left i.s mainly errors of measurement. Stake, ^ basing In'.s calcvilatlona 
ou typical values for the reliability and standard deviation of gain scores, 

i 

dciicribcd wliat could hoppen in a hypoihctlcal performance contracting stliua-^ 
tion . 

Suppose that 100 r.tudonti; were admitted to contract iiivStruction 
and pretested. After a period of time involving no training, they 
were tested again and the students gc\ining a year were graduated. 

After another period of time, another test and another graduation. 

After the fourth terminal testing, even though no instruction had 
occurred, the chances are better than 50:50 that two-thirds of the 

\ 

students would have been graduated. 

'fhe unreliability of gain scores suggests that they should not be used 
to differentiate; students who have learned more from those who have learned 
less. It does not mean that differences befween average pretest and average 
post-test scores are also unsatisfactoi'v as Indicators of instructional 
effectiveness. Hie standard error of a mean of test scores is much smaller 
than the standard error of an individual test score. In assessing the 
perf on.iance of an in.s tructional process, it is much better to use class 
average gains than individual pupil gains as a basis. 

There Is, of course, a danger in focusing attention on class average 
gains. It is that a teacher eager to sliow maximum gains might concentrate 
effort.s on the ablest learners in the cla-.s, neglecting the slower learners. 

^ Stake, Robert R. "Testing Hazards in Pcrfonaance Contracting,” Phi Delta 
Kappan , 3.?;563-300, June, 1971. 



llc/.7 well such a sl:nitc|;y would pay c.fr is opcMi to quo.silon. How c.aflily a 
lc;K.li(*r could cKf ferenLlaU* her cHorts tiriion^ atmlrntr* is alno open to quos~ 
tlon, ConscicMUlous teachers would no doubt disdain to experiment with it. 

If pupils. are sufficiently alike in learning, ability to belong in the same 
class, the best strategy, and the (nie most good teachers would be likely to 
follo'W, would seen) to be to give oc|ual attention to the instruction of eac:h 
3 tudent , 

Kclatcd to the question of basing assessments on gain scores is the 
question of taking account of pupil learning abilities in assessing educn-- 
tional perfomances. Previous acliieveinent in learning, whether indicated by 
ncorcs on achievement tests or by scores on scholastic aptitude or intellL- 
gcnco tests, is probably the best indication of learning abilities, llus Is 
so* because previous achievements indicate the student's ”track record” as a 
learner, and be<^aur,e they indicate how solid and extensive his foundation for 
further learning may be. 

To e.>:pcct a student vn th belov; average learning ability to gain as much 

frcei a particular instructional program as another student with above average 

learning ability is clearly unreasotu j le , But, while learn.xng ability is an 

important daterminer of success in future learning, v;e should guard against 

the notion that it is the sole determiner. For learning is an individual 

\ 

achievement v;hich calls for considerable effort that only the learner himself 
can make. If, for any of a number of reasons, he chooses not to make the 
effort, or finds it unduly difficult to make, he. v;ill surely succeed badly 
in future learning. The important dimensions of the learner's dispo.sltlon 
to learn and his willingness to work hard at the task arc soaiGtimcs overlooked 
or disregarded in sonje of the newer instructional technoioglci; and instruc- 
tional programs. 
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SunO.y Llio Ic'anilii;^ abllltici of the puiiils do ncod to be taken Into 
account Jn aa.se.s.': i nj; tin; educational pcrforiaancus of pupils and tlielr 
instructors. It is usually unvs^Jse to use a sinpjo. IQ score as a sufficient 
measure of leaniinj; ability. It is hazardous to assess the educational 

I 

pe rf onaances of individual pupils in relation to their inferred learning 
abilities. The sane kinds of errors that afflict individual gain scores, 
and some otliers, afflict estimates of individual performance in relation to 
ability. Achievement quotients were discredited and discarded long ago, but 
the came notion, involving the same errors, persists when some students are 
identified as underachievers and others as overachievers. But, when wo 
assess the performance in learning of classes, in relation to the level of 
learning ability of the class, we are on firmer ground. 

8. V.’hat level of performance should bo considered satisfactory? 

t’ne edi:c.ational performance of students is assessed to determine 
the amount of their learning, or to determine the effectiveness of their 
instructors in teaching, the results may vary from somewhere close to perfec- 
tion to somewhere close to zero. The performance being assessed is a contin- 
uous v.irieble. Often the assessments can be expressed in nambers, so that 
the larger of t^^'o nirnnhers represents the greater amount of learning or the 
greater effectiveness of instruction. Ihesc numbers provide information that 
has a variety of uses, simply as descriptive information. 

Sometimes, however, categorical decisions must be made. Is this amount 
of learning, or that degree of effectiveness of instruction satisfactory? 
Students may be dropped, applicants may be denied certification, teachers may 
be retained or dismissed on tlie basis of such categorical decisions. Tlie 



clot:(-‘iinln.it l(‘n of wli.u lovol of juM' forni;mco J:.* Lo bo ro[;ardccl a:j i;atl/;fnct:ory 
In any pnrllcuK'u* slLualion Is a con’plox and dlfflcnlL problem, No nlmpio, 
p/jiuirally effeclfve procedure for ruUvlnj; .‘^ucli prol)leiiis v/J.11 be ted 

here, Probiibly no such procedure cxlrts or can be developed. Kliat can bo 
done is Lo su^ge.'iC certain yiildel Ines that, may facilitate discovery of an 
acceptable , th(Hi/;li 1 nev t tab ly i i:\per f cct , solution. 

One of tliesc» obvious cnou/'^li but son;etlmon denied, is that couiplote 
nastory in le.an.in^ any but the siir.plest per:^oniiancos , or total effectiveness 
in instruction, is impossible. One nay be the co^nplete master of spelling, a 
particular word, but no one ever completely masters the spelling of all 
Pngllsh words. Perfection may be useful as .in ideal goal but wc do well to 
recognivie that it is in fact, an unattainable goal. Those v;ho purport to 
teach for mastery always are willing to settle for something less than 
complete mastcr>’. Either they tolerate some degree of error or inipcrf ect ion 
in the porf cmar.ces oi their pupils, or they define such specific enid limited 
goals that partial and incomplete knowledge can put on the disguise of 
complete mastery. 

Another guideline is that completely objective, empirical determination 
of a minimum level of performance to bo called satisfactory is out of tlie 
question. Subjective judgment and arbitrary declarations will alv;ays be 
involved. More of learning or skill la instruction is alv;ays better than 
less, but there in no point on che scale below which performances are demon- 
strab3.y unsatisfactory and above which they arc demons trrb ly scUis factory . 

Ti\o point is fixed by a declarative Judgment. Its location is detenained 
entirely by the value;; and standards of the judges v;ho establish it. 




A tlilrd i;ui(lc‘l I no is llwU: norin;i c i vc ]y bcisocl MnnOarcln aro likely Lo be 
more de fcMi.s lb le Oian cc)iUent relaled sLandaids, A standard Ls normailvely 
baj-.ed if it Is located so that a specified proportion of tl^c IndlvlJualH in 
a r> j)C.‘ci n c'd ^roup exceed or fall bclw it. A standard 1:3 contenu related If 
It specifioy vlial proportion of tlie tasl:s In a particular i;i*oup of tasks ir.uat 
be successfully c:«)!r,pleted. IJxcc 1. 1 cnce and deficiency In a human perfonnanco 
Jn usually jud[;o.d relative to otlicr performances. If one can do as well as 
or better tlian the average performer, hir, performance Is likely to be rcp,ardcd 
as satisfactory. Most business -men vould not consider a 20-word per minute 
typist satis fcictot')' because it is easy to find typists wbo can type at three 
tines that rate.' Neither a standard v:hic!i nobody meets, or which everybody 
meets (and some far exceed) is likely to be regarded as a satisfactory 
standard. Seldom is it possible to find a rational basis for setting a 
content-related standard. 

Tl^ese guidelines may be of some small help in determining a satisfactory 
level of perf ornance, although they leave much of the problem to the knov;lcdge 
and judgment of tho^^e v;lio need to set stcindards of performance. Fortunately, 
the need to differentiate categorically' a satisfactory fran cfn v-nsatisf actory 
performance does not arise in most assessments of performance, 

9. Some recon>nendations • 

What has been said in this paper can be summarised in the follov;lng set 
of rccoimendations : 

1. Recognize the fact that educational performance can he assessed, but 
that the process is neither simple nor easily managed. 

2. Regard the perfonuances being assessed not as ends in tbomisclves but 
rather as indicators of developed competencies. 
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3# Acct‘])L llio cultivation of co^»nlLivo coi:i;) 0 tonrc an l\w. innlu. objccll.vc 
of f.dioo 1 i > with the dew loiuncui l of affective disposition?? as impor- 
tant byproducts. 

A. Define coj*nitivo competence as acquisition of knas'led^ic rather than 
as dew JLO/if:on t of 1 n t •* lloct .n:H skills. 

5. Rocojpuao the value of objcctlvo tests in tlie assessment of perfor- 
mance , 

6. Note that a fully inoaninp.ful test score carries botli nonnativc and 
content ^'elated meaning. 

7. Observe that tlie difference between a good norm-referenced test and 
a good criterion referenced test is greater in theory than in practice. 

8. Rccognijce tlie practical impossibility of complete mastery of any but 
the simplest, most specific elements of knowledge. 

9. Recognize the practical impossibility of obtaining test score.v which 
are directly proportional to the amount learned. 

10. Do not expect tlie use of criterion referenced tests to bring about 
substantial improvements in assessing educational performance. 

11. Recognize the desirability or accept the inevitability of external 
evalua. tions of the effectiveness oC educational programs. 

12. Seek relevance in the tests used in external evaluations, and 
expertness and impartiality in the testers. 

13. Give preference to a custom-made test over a pijblished standardized 
test for an external assessment of educationax performance. 

lA. Guard again:2t teaching to the test by avoiding published tests or 
re-use o^ specially constructed tests, or by warning teachers and 
]'« T-f orii'.a-. • contractors against it, and a5;king pupils and supervisors 



to report it. 



15. Uio clan^ r.Ul'ui' Lli.m lmlivit.lu.il pupil scores to measure 

gains frun In'iLiuction, or to take account of clif ferunce.s in le/in»lng 
ability. 

16. Recognise the impos.''>ibi 11 ty of leachir.g to complete mastery or of 
co.’.'.pletcly objective dc term in.it ion of .a minimum .satisfactory level of 
performance. 

17. Roly more on normati.vely defined than on content defined standards 



of acceptable pe rfori.nance . 
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INTRODUCTTQN 

Any exnmin.'i t i on of altcrnal.ivp. cclucnlional prop, ranis must: be con- 
cerned v/ith their ef f ect i venesr, and cost. Jiecause student performance 
is one of the measures of the effectiveness- cf the propra^i, a preat 
deal of attention is being given to the prcd')lems of setting criteria 
of achievement and measuring educational outcome. Less attention has 
been paid to the equally demanding task of estimating and analyzing the 
cost of educational programs. If the instructional strategy of new 
programs is to be successfully utilised by educational planners, infor- 
mation about the cost as well as the effectiveness must be available 
to the decisionmaker. 

This paper explores the conc.epcuai and methodological basis of cost 
analysis and develops a planning cost model for estimating program cost 
for use in evaluating alternative programs and in pre-implementation 
planning for future programs. The planning cost model with its support- 
ing cost analysis methodology provides a consistent basis for esti- 
mating the dollar cost of educational progiams. The development of the 
model was undertaken because the current state of the art in costing 
educational programs docs not provide a comparable basis for evaluating 
alternative programs, 'fhe usual practice Is to give the cost per stu- 
dent for a program with no indication of v;hat is included in the cost. 

Wlicn the cost per unit of achievement is used, both the cost and 
the effectiveness meas ircmcnt problems are severe. Education Turnkey 
tiaWs has drawn attention to several aspects of using this ratio: 



Any views expressed in this paper arc those of the author. They 
should not be Interpreted as reflecting the view of The Rand Corporaticni 
or the official op:lnion or policy of any of its governmenta] or private 
research sponsors. Papers are reproduced by The Rand Corporation a55 a 
courtesy to members of its staff. 

This pnjH'r v/as prepared for prcscritntlcui at the National Conference 
on Performance Contracting in luhnation, Washington, D.C., Dcuu^nher 9 
and 10. 1971. The Conference is cn-sj)onsore(i by the Amoriewan liduca t Iona I 
Rtusearen A.ssociatlon (AKRA) and the American Association of S(‘Iiool Admin- 
istrators (AASA). . 
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Kvon wlu‘n arr*iiratc costs are oht;niied, it; is difficult to com- 
pare ti)om with school costs to see vdiich is less, since school 
costs arc kept and reported di f fc’rcMU ly , The comparisons may 
reveal, nothing, more than different fijun'os, ejjpecially since 
th(‘ firms [piM'fcrinance contractors in the context of this quo- 
tation) may depreciate certain itc'ms much more rapidly than 
schools It is even more difficult to try to contrast ef- 

fectiveness witli cost. If effectiveness is reported in tenth? 
of a year’s achievement, which some statisticians feel is cut-- 
ting it too closely, and that figure is divided into cost data 
which is part hidden and part hypothetical, what does the pub- 
lic get? Will a school hoard really base a major decision on 
curricular changes on such a ’’cost por unit of achievement” 
f igurc?**^ 



The ratios cf cost per student and of cost per unit of achievement are 
widely used, probably because of the false confidence the ’’number” en- 
genders and the relative ease v;ith which it can be generated. In most 
instances, cither ratio masquerades as the output of cost-effectiveness 
analysis. Wisely used, cost-effectiveness analysis of educational pro- 
grams produces several outputs — the aspects of cost, the measures of 
effectiveness, end the r*clationshipa hetveer, coot and cffactivcnacs , 

The problems and the appropriate use of cost-effectiveness analysis in 
educational planning have been discussed in Cost-Effectiveness Analysis 
fop Educational Planning, Only very seldom is a ratio of cost per stu- 
dent or cost per unit achievement the appropriate end result of a cost- 
effectiveness an^i] ysis . 

The planning cost model and its supporting methodology of educa- 
tional program cost analysis provide a solid basis for resolving, at 
least in part, the problems encountered in determining the ’’cost” of 
educational programs . The planning cost model assists in developing 
comparable cost estimates of alternative programs. In this way, the 
model directly addresses the problems inherent in using an undefined 
cost per student in evaluation of different programs. 

In estimating the program cost to he used in conpaving programs , 
tl;3 resources available within a specific district or assets inherited 



Reed Martin and Peter liriggs. Education Turnkey News^ Fc.bruary- 
March 1971, 

'^Cost-Effectiveness A>ialysis fo)'* Educatioial Planning^ M. B. Car- 
penter and liaggart , The Rand Corporation, P-4327, March 20, 1970; 
also reprinted in Educational Technology ,, October 1970, pp. 26-30. 
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from (liscont: i mio.cl proj»,rains nrc not taken into account, and n jitnndard 
price for commcn resources, such ns teachers, Is used. The result Inp, 
estimated proj»rani cost is identified as the conparcxhlc vcpliaatio>\ cost. 

It is. In essence, a comparable cost that normal i;:cs the cost of proj;rams. 

In estJr.ntinp, the prop, ram cosl to be used In deciding’ whet.ier or 
not a particular program can be implemented in a specific district, the 
resources nvnLlablc v;ithin the district and district-specific prices 
for these resources must both bo determined. The resulting estimated 
program cost Ln this case is the increnioitol cost to the district. 

The role of the planning cost model in estimating both the compa- 
rable replication cost and the incremental cost is pictured in Fig. 1. 

In this process, the first step, common to estimating either the compa- 
rable replication cost or the incremental cost, is a definition of the 
program in terms of its objectives, its students, and its resource re- 
quirements. These resource requirements are translated into the type 
of program cost estimate relevant to the decision to be made. The plan- 
ning cost mod^l, by providing a o.onsistent methodology for estimating 
program cost, helps insure cost comparability among programs for deci- 
sionmaking purposes. 

Before describing the planning cost model, a short discussion of 
the concepts and techniques of cost analysis underlying the development 
of the model should be helpful. The use of the model in estimating 
the comparable replication cost and the incremental cost is illustrated 
in the final part. 

COST ANALYSTS 

Cost analysis is concerned with the determination of physical re- 
source requirements for the program, with calculating the program dollar 
cost, and with systematically evaluating the impact of changes in the 
program on both the resources needed and their dollar cost. The ap- 
proach is to first determine the facilitl"' , staff, equipment, materi- 
als, and services needed to conduct the educational program and to then 
translate these resource requirements into an estimated program cost. 

This sequence forces explicit consideration of the varying resource re- 
quirements for different programs or for changes in program scope. 
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Fig. 1 — Process cf estimofing the corrparoble reriicntion cost and :he mcrenentol cost of programs 



TIk* odnr at i ona 1 pr(>^»ram has as Iti.s core an 1 nst rut’f tonal siratop.Vi 
This instructional r;tratca»y inrlmles botli tlir. rf£*.our((*s and tlic way in 
wliich tin* rosourcor. arc used to produce tin* (uiuratlonal outconu*. 

IV f i an i’* . 

Tlie first" stoj) in analyzing, the resource requiretnonts and cost of 
a pro';ram is the definition of the prop.raiiu The quality of the estimate 
of the cost of an educational prop.rarn depends on the conipl etenoss with 
which the resource requirements of the proj;;rem arc determined. Tliis 
determination, in turn, depends on th? description of the educational 
prop,rem. ITie sequence of events then begins with a description of what 
the program is and how the program works and continues with a determina-^ 
tion of the quality and quantity of the resources. These resource re- 
quirements arc translated into an estimate o^’ the program dollar cost. 

In defining the program, the types end magnitude of support activities 
or services also need to be identified. 

Determination of Resource Requirements 

The definition of the educational program is followed by the dater- 
tnination of the resource requirements, The data required are arrayed 
in the illustrative format of Fig. 2, Some of the categories in Fig, 2 
pertain to resources directly. Others are "functional packages," such 
as training, whicli are combinations of resource items. Additional data 
should be provided as appropriate for specific programs. Each of the 
items in the format will be defined in terms of the kind of information 
needed . 

Data about the characteristics of tl^e students served and the num- 
ber of students in the program will, of course, be the same data required 
for the evaluation of ihc. effectiveness of the. program. Data on other 
district conditions tliat might have an effect on the outcome, such as 
income level, turnover rale, or mobility, sl’.ould be provided. The in- 
structional time shoiJd be given, along witli other Information tliat re- 
lates to determining the actual time spent witli subgroups of students 
or individual students. Tl\c stiident -teacher ratio is usually used as 
a proxy for ibis, but an (Tfort sl\o\ild be made to refine this riecc of 
Information. r y 
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Character is tics of StudcMit.s Served 

Number of Stuclc?nts 

Instruct ional Data 
Clns.f t ime 
Class size 

Facilities 

Space 

Students/cl assroom/day 

Utilization 

Furnishings 

Staffing 

Teachers 

Special teachers 
Paraprof essionals 
Other personnel 

Equipment 

Program-related 

Student-related 

Materials 

Program-related 

Student-related 

Pre-service Training 

In-service Training 

Other Support 



Fig. 2 — Format for progrevn ard riASjuroe information 



In describing the facilities needed, the space requirements. In- 
cluding mobile or portable classi'ooms, laboratories, and their utiliza- 
tion rates, should be carefully determined. The requirements for non- 
Bchool facilities should also be stated. The special needs for electrical 
outlets, air conditioning, carpeting, and lighting should be Identified. 
Furniture needs are to be specified. Identifying any special per-student 
requirements . 
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Stnffjn/» for iho i)rop,r/nn .should ho tlc.scr ibod jn tonns of llu* (ju.nli- 
ficntloiu; needod as well as in terms of number (e.f;., r,ivo nuinbi»r of 
cerllf icat (fd or certiflod t eacbors, the number of s]>ecial teaclu»rs, 
paraprof (»5;s ional staff, and other personnel involved in the prop, ram) . 

If a staff member works less than full time, the percent of time in- 
volved should he piven. Staff requirements for time beyond the "normal** 
school day should be stated. This includes, for example, custodial or 
security services needed to keep the school open after the rcp.ular day, 

Equipment and materials should be identified as program -related , 
classroom-related, or student-related. Program-related equipment or 
material is that which will be used by several students d’-ring the day 
or some tine period of the program. Very often the equipment or mate- 
rials may be grouped by classroom unit. Student-related equipment or 
material is that which is required because there is a specific number 
of students in the program. An additional distinction should be made 
about the consummable nature of the materials and about the lifetime 
of the equipment. The same treatment shou-.d be applied to supplies if 
the usual disirict practice is tj treat equipment and supplies as sep- 
aralt* categorler?. 

•The amount of time involved in pre-service and in-service train- 
ing should be specified. The materials or equipment required should 
be given. It should be noted if the training time is incluled as part 
of the regular time of the staff or if it is incremental to the regular 
Working hours. If in-service training time is a substantial part of the 
Individual teacher’s time, addit:onnl teachers (or substitute teachers) 
may be required for the instructional load of the program. 

The requirement for program-related services su:h as evaluation 
or other management activities should be given. It is preferable if 
the actual time or the numbers of consultants can be specified. In 
either case, the purpose is to provide some estimate of the magnitude 
of these services so that the decision can be made on whn t it costs to 
buy the scrviLC rather than to develop, if possible, an in-houso capability. 

Support from other activities means the support required by the 
educational program from such service functions as transportation. For 
example, a particular educational program might need bus transportation 
for field trips. This instructional ly-requlrcd traiisportation is over 
and above the cost of homc-to-school transportation. 
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Th(‘ rt^soiircc rccpil rcnicnLs idon t i f i t‘d in ?• nro innnnt to be snj‘,- 

(;osl Ive only. If other clnfa are available, they should be j\lven, since 
tlu* purpose is to define as completely as possili^t? those resources and 
cos t-p,(MU*r • l. i np, activities needed ro carry out the educational prop, ram. 

The resource requirements are (hen translaled into the dollar csti-- 
mates of propram cost — either the coinparable replication cost or the 
incremental cost. A planninp cost model provides a framevork for sys- 
tematically and consistently iFtlmatinp program cost. 

THI:: PLANNING COST MOD EL 

The planning cost model provides the: mechanism to determine, con- 
veniently and consistently, the cost of various alternative programs. 

By design, the model IvS appropriate for pencll-and-paper operation as 

* 

well as computer operation. 

The model provides the framevork for bringing together the resources 
(facilities, staff, equipment, materials) required to carry out an edu- 
cational program and for relating these resources to program output ip 
the form of activities. 

By relating the Inputs required to produce outputs, in terms of 
act^^ities, the model provlcies more Information for making decisions 
about the merit of selected changes in the activity structure of the 
total program. For example, trade-offs between fewer but longer instruc- 
tional periods and more but shorter periods could be assessed. The 
model also provides the basis for examining the cost consequences, for 
the total program, of changes in the resource utilization rate (l.e., 
student/teachcr ratio) or in resource cost (l.e., teacher salary). 

The task of constructing the model demauucd a close examination 
of the concepts of cost analysis , especially in their application to 
educational program cost methodology. This examination resulted in the 
delineation of an approach to costing educational programs. Basic to 
this is the definition of a preliminary list of cost categories. Those 
costs of school district operation not affected by the existence of the 



A planning cost model designed for computer operation is described 
in R-672-SJS, Projaat San California: Evaluation of RgcuHg 

and Development of a Coot Model, M. L. Rapp, M. B. Carpenter, S. A. Ilaggart, 
S. 11. Lancia, and G. C. Si’ nor, Tne Rand Corporation, March 1971# 
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proj’ram arc not Inchick^l in tiu* estim.Jtccl cost of tbo prop, ram. An ex- 
ample will sorv;o to clarify this point. 

The district cost catep,ory, transj^ortation , provides for the trans- 
portation of students to and from school. Fcudenls in the special pro- 
gram will contiiuH to receive ti insportation, if they need It, jiist as 
thougli they were not in the special program hut were, Instead, students 
in the regular program. This regular transportation cost is not included 
in the cost of the individual program. But, if tlic instructional method 
of the special program calls for field trips or other activities lequiv - 
ing t.r* tspert.ition , the cost of this transportation is included as a 
cost of the special program. 



Cost Categories 

The iten.s, services people, and activities and their cost required 
for an educational program can be brought together In one format — the 
cost element structure shown in Fig. 3. Tnese cost elements are grov^ 
into two brocad categories: the acquisition cost and the operational 

cost. The cost of most programs can be adequately encompassed within 



Acquisition Cost 

"k 

Design of program 
Development of materials 
Evaluation design* 
Program implementation 
Equipment 

Program-related 
Student-related 
Materials and supplies 
Program-related 
Student-related 
Prc-service training 
Facili tics (space) 
Installation 



Operational Cost 

k 

Program direction 
Evaluation* ^ 

Management support 
Salaries 
Teachers 

Paraprofessionals 

Specialists 

Other 

In-service training 
Materials and supplies 
Program-related 
Student-relatci. 
Eqiilnment 
Replacement 
Maintenance 
Facilitier, 06M 
Contracted services 
Media services 
Transportation 



In an oj>c^rational program, as opposed to n demonstration 
program, there might be no progrt^m cost associated with these 
dcLivitlcs . 

Fig^ 3 — Cost elment struoluve for educational programs 
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t;lu*s(‘ two bi’o.'ul cntcj’,or1(’s. The.* n{*(iuir>llion cost Is the ono'-limo cost 
to nccjuiro a capability. The operational cost Is the conMmilnp, cost 
to maintain a capability over a period of tine. In the followlnp, dis- 
cussion, one ytiarVs operating cost is assumed. 

The acquisition, or one-tlrne, cost to acquire a capability is, in 
practice, also referred to as initial, investment, or capital cost. It 
covers the cosi of all resources required to acquire a capability. The 
cost of the effort devoted to resnarcli, development, or dosipn of com- 
ponents of the program or alternative's should be included as part of 
this cost. The cost of desipning a different mathematics curriculum, 
for example, is a development cost. In estimating the aofrparable / 

lication cc*st, however, some overall, development costs might be treated 
as sunk costs. That is, the first program to use the new curriculum 
would incur this expense, and subsequent programs using the curriculum 
would inherit the new curriculum on a cost-free basis. On the other 
hand, if the curriculum had to be redesigned for a patticular program^ 
this would be a development cost for that program. 

The operational cost is also I'eferred to as the recurring or con- 
tinuing cost to maintain the capability. The cost ot modification of 
facilities and the cost of in-service training of teachers are included 
as an operational cost to maintain the program. These broad categories 
of cost — acquisition and operational — are used as a basis for organizing 
the cost elements into the cost element structure. 

This structure provides the framework for identifying the cost of 
the program in an operational environment. Each element , whether it 
is an item purchased or an estimate of activity cost, will be discussed. 
But first, remember that costs not varying because of the existence of 
the program arc not included. For example, district-wide administrative 
costs are not allocated. 

Costs that might be incurred in a demonstration program but not in 
an operational program are identified by an asterisk in Fig. 3. Some 
of the cost categories can be characterized as the cost of activities 
rather than the cost of items purchased. In many instances, the items 
purchased quite clearly underlie the cost of activities, but the activ- 
ity cost, however, may be used directly in estimating the program cost. 
For example, the evaluat ion cost of a program might bo. estimated by 
using a factor such as cost per student. Or, the cost per program might 

iG 
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If Llio ovahiaLlon i:\ done by an oulslcln roiUT«iclor or rvalnalor. 
«'j])propr lalo , Ihrse wu\i]d bc! Iho factors used to ostiinalc tbe opera- 
tional cost of evaluation* The aoquiGitio>J cost — the non-reeurrlnp, 
cost — for evaluation mifdit be based on the district staff time to derilr,n 
tlie evaluation of the pro['rain or mip,ht simply be tlu* cost cbarj»ed by the 
outside evaluator, llie cost basis for these inputs would be per projrvn 
for acquisition cost and per student or pvogrcun for the operational cost. 



Cost, basis for Inputs 

The cost basis for all inputs for the categories in the cost ele- 
ment structure is shown in Table 1. For each category the cost' basis 



Cateejories 

Acquisition Cost 
Design of program 
Development of materials 
Evaluation design 
Program implementation 
Equipment 
‘ Progran- related 
Student-related 
Materials 

Progvan- related 
Student- related 
Pre-service training 
Facilities 
Installation 

Operational Cost 
Program direction 
Evaluation 
Management support 
Salaries 
Teachers 

Paraprofessionals 

Specialists 

Other 

In-servl ce training 
Materials and supplies 
Program-related 
Student- related 
Equipment 
Replacement 
Maintencince 
Facilities O&M 
Contracted services 
Media services 
Transportation 



Service 



X X 

X 



X 

X X 

X 

X 

X 

X X 

X X 

X X 

X X 

X 

X 

X- 

X 

X ‘ X 

X X 



Table 1 

THE COST BASIS FOR INPUTS 

Cost Basis 



Studen t Program Uni t 



X 

X 

X 

X 



X 



X 
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is oilluM' [)‘T student, per pr(?)»rnm, por unit, or (Urort service cliarp,e. 
The per student niul per ])roj;rnni distinction Is r*illu*r obvious; tlie por 
unit basis r(*fcM's to units such as classrooms, resource centers, and 
lanf;ua)»e laboratories. The service basis is used when the input to the 
model mipdit be the extent of a service performed cither within the dis- 
trict: or by an outside source. An example of the former would be the 
operation and maintenance of tlie facilities; the latter service-based 
input mipjit cover such items as the. contracted transportation for the 
instructional part of a program or the provision of so many hours of 
instructional television . 

In some cases, the cost input basis might be a combination of pro- 
gram and unit (classroom), of student and service, or of program and 
service. No rigidity is implied. The intent is to provide an under- 
standing of how the inputs of the model are categorized. This catego- 
rization is basic to the structure of the planning cost model . At this 
time, it is only necessary to emphasize that some level of input is re- 
quired because there is a certain number of students, and other levels 
of input are required because there is a certain number of classrooms 
or instructional centers. In many cases, there is a program cost that 
Is independent of the number of students or centers. 

Outputs and Inputs of the Model 

A program-related cost can be a thruput to the model. For example, 
the cost of program development wculd be both an input and output. The 
cost of pre-service training for the teachers in the program is calcu- 
lated within the model. The physical descriptors of the program and 
cost factors, such as the number of teachers, the salary cost, the cost 
per mile, are the inputs to the model. The objective is to keep the 
number of inputs to a workable minimum while allov/ing enough input flex- 
ibility to provide useful outputs of the model for the evaluation and 
planning of educational programs . 

Tlie outputs of the model are, in general, the resource and cost 
information about the specific educational program. The descriptors 
of the prograrr , — number of teachers; number of students; space require- 
ments; equipment, materials, and supplies; and need for services such 
as transportation or evaluation — are shown right along with the cost 
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out'put. llio pnrposo is to provide, in one place, an estimate' of the 
comparable replicalion cost and a dc'scrlpllon of wbat ts beinj’ boup,bt. 
As this pi art Ice becomes more prevalent, the use of a cost per student 
to describe nn unknown quantity will decrease and the quality of infor- 
mation available to (be educa t ional planner will increase. 

The output of the model Is Illustrated In Fit's. 4 and 5. Notice 
the similarity of the format to the cost element structure of Flp. . 3. 
More detailed info7*mation for any of the Items shown can be provided 
in supporting reports. For example, the resources and cost underlying 
the cost per student hour under Media Services might be of interest for 
some typos of decisions. The r>upporting detail for this would follow 
the same cost element structure used for estimating the cost of the en- 
tire educational program. 



Description of Program 

Program: 

Staffing: 



Objective : 

Student Characteristics : 



Facilities : 
Equipment : 



Materials : 



Acquisition Cost 

Prog ram activities 
Equipment 
Facilities 
Materials 

Total acquisition cos t 

Operational Cost 

Program activities 

Salaries 

Materials 

Supplies 

Equipment 

Other support 

Total operational cost 



Operational Characteristics . 
Instructional time 
Student grouping 
Location 



$ XXX 
XX 
XX 
XX 

$ xxxx 



$ XXX 

xxxx 

XX 

XX 

XX 

XX 

* $xxxxx 



Fig. 4-^Swrimavy oi*tpuL of modal 
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Other support: 



Prop.r.’iiii ArLiviiios: 


I)c>.sl{',ii of rrop,rain 


X 

X 

X 

</> 




Dovel 0|)Iik:iiL of MaLoti/ils 


XXX 




Kvaluati on Dcsic.n 


XXX 




Program ]i:i|)lniiciiLat ion 


xxxx 




Pre-service Training 


XXX 




Installation 


XXX 


LquipmeiU : 


Program-related 


X 

X 

X 




Student-related 


XXX 


Facilities : 


Student-related 


</> 

X 

X 

X 


Materials : 


Program-related 


$ XX 




Student-related 


XX 




Total Acquisition Cost 




perational Cost 


Program Activities: 


Program Direction 


$ XXX 




Eva] uati on 


XXX 




Management Support 


XX 




In-service Training 


XX 




Facilities O^.M 


XX 




Contracted Services 


XX 




Media Services 


XX 




Transportation 


XX 


Salaries: 


Teachers 


xxxx 




Specialists 


XXX 




Paraprofessionals 


XXX 




Other 


XXX 


Materials : 


Program 


XX 




Student 


XX 


Supplies: 


Program 


XX 




Student 


XX 


Equipment: 


Replacement 


XX 




Maintenance 


XX 



$ xxxx 



xxxx 



XX 



XX 

XX 



Total Opcratxonal Cost 

Fig, 5 — De taxied output of program cost estimate 
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Tlu* of t.lir* fol) Into throo bro.ul |;ronp5;i (1) iho 

physical lU'Pcrlplors of tlio prop, ram; (2) Iho rosi of rosion rcj*. and ser- 
vices; and (3) Uu* factors or es 1 1 ma 1 1 i\p re 1 at lot^sl^ 1 pf; . The phy;>ieal 
(Icscr iptur.s , including, (ho type ami qiiantftv of resources, were shown 
in Fip,. 2, for I'rnjVin :ikI Hcco\<roc Infcrnitio^i. In short, tiu?se 

inputs d('scrlbe tlie students, the educintional propriani, land the resource 
requirements. Inputs are required for all the chanj»i*s, or variables, 
that make one propram different from another program. 

Tlu! Inputs describe the cost of resources and services and cover 
such items ns the cost of equipment used, the salaries of the staff, 
the cost of testing, the cost of transportation, and the cost of train- 
ing. The input factors, or estimating relationships, include both cost 
factors such as cost of materials per student and non-cost estimating 
relationships such as number of in-service training days per teacher. 
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The Structure of the Model 

The model integrates the program description, in terms of resources 
required, with the process of estimating the program cost. This process 
begins with (he determination of resource requirements and continues 
with the translation of these resource requirements into an estimate 
of dollar cost. Both the acquisition cost and the operational cost are 
estimated . 

The model’s framework for estimating the acquisition and the op- 
erational cost is shown In Figs. 6 and 7, respectively. For each cost 
category there is an estimate of cost on either a student, program, unit, 
or service basis. In the case of ’’units,” the estimate can be the cost 
per teacher, the cost of the equipment per classroom or instructional 
center, or the cost per student or materials consumed. For some cost 
categories, the estimate can be based on an overall program cost. For 
example, the pre-service training, If done by an outside contractor, 
might be a total cost for the program. It could also be a cost per teacher, 

In the cost category for Mtiterials, the cost estimate may require 
an estimate for the cost for student-related materials, for the cost 
of materials in the classroom for use by many students, and for the cost 
of program materials used by the staff in conducting the program. Tlie 
same practice is followed for the cost categories of the frnmev’ork for 
the operational cost in Fig. 7. 
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Fig. 6 — The planning cost model — acquisition cos 
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Tlu» ro.sl: Ciitc|;orlos provldo a convonlcnt wny to Idf^ntlfy the data 
n(?cdi'd ahout ihc* cdurat ional proj;ram and Iih operation In order to ea- 
tlmaio its cost:. The data for tlic cost catop,orlcs for both the ac'iul- 
sltion and operational cost are shown separately in Flp,K , 8 and 9, 
vesper I i vely . 

USK OK Til l: IM^NNINC C OST MODKL 

Kstlinal l np, the Com p arable Re pi lent ion Cost 

The use of the model will be illustrate d by estimating the compa- 
rable replication cost for several different programs. It should be 
emphasized that in order to compare programs in different districts, 
comparable resources prices and salaries have to be used. A comparison 
of actual costr> would have little, meaning since the differences among 
programs would not only reflect differences in the programs but also 
differences in teacher salaries and other local prices. 

As shown in Fig. 1, the process of estimating the comparable rep- 
lication cost and the incremental cost for a program begins with a de- 
scription of the program and its resource requirements. This informa- 
tion is then processed through the model in order to estimate the cost. 
The description of the program includes both program information and 
resource information as shown in the format of Fig. 2. 

The program and resource data for several Illustrative programs 
are given in detail in the appendix. The summary of this information 
is given in Table 2. The resource requirements are estimates of what 
It would take to replicate the Instructional strategy of the program. 

The information under Other Support provides an example. In the 
replicated program, there is an item for consultants to the program. 

It is estimated as approximately eight days for the year of program op- 
eration. This is an estimate of what might be needed in a future pro- 
gram rather than an estimate of what was used in past programs. The 
same is true for Program Evaluation. A category for this type of actlv 
ity calls attention to the need for evaluation of the program even in 
operation as part of the regular district programs. In the estimate 
for the replication cost, this tegory Incurs a cost per student for 
evaluation of the program. 

IC/'. 
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Cod Cafe JO VII 

D(*f;lp,n of rrc));r;im 
Dc vol opncTit o r Material 
Kval nat i on Dos i p,n 
Prof^rnm Imp 1 (»nr/rua t ion 
Pro~‘iorvice Training 
Inslullation 

KqulpmcnC 



Facili ties 



Materials 



Data l\CA{ui7\'^f^',ania 

Tf thc:;e activities arc required for the 
program, the mcdc)\ the of personnel 

involved, the tine spent, and fjalary arc 
needed. 



The equipment list is determined for each 
student, for each classroom, and, if appli- 
cable, for the program, llic classroom's 
equipment is used by several rla.sscs of 
students. The number of students that can 
use the equipment is specified. 

The space required is that over and above 
the regular program; both for each student 
or for special resource centers. 

The initical stock of materials is deter- 
mined for each student, for each classroom, 
and, if applicable, for the program. 



Fig. 8 — Vvogvem data — acquisition cost categories 



Cost Category 



Data Requirements 



Program Direction 
Evaluation 
lUmagement Support 

Salaries (with fringe 
benefits) 



Materials and Supplies 
Equipment 



Facilities O&H 
Contracted Services 
Media Services 
Transportation 



o The number and type of staff, the time spent 
for each activity, and salary are needed for 
this. 



o All instructional staff and direct support 
classes of staff are identified by broad 
category; i.e. , general teachers, special- 
ists, and aides rather than a teacher with a 
specific salary arc used. Fringe benefits 
are included at the district percentage 
factor. 

o The type and quantity of materiali used are 
specified on a student and program basis. 

o The equipment maintenance factor and the 
equipment replacement factor (based on the 
estimated lifetime of the equipment) are 
applied to the equipment used in the program. 

o The program requirements for each of the 
categories are specified in terms of square 
feet maintained, services purchased, number 
of hours of audio-visual instruction and 
bus trip mileage. 



Fig, 9 — Program data — operational cost categories 
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PROGRAM Kr.SOURGi; RliOUlRKMI’-MS 



Itcf^ 




/Tj.mt'J r 


c 


rr.J jr\:n 0 


Prejnn V 


Vr:*^r n f‘ 


HuTiicr of Sludiiit* : Rrjdl ij 


)V0 


28*> 


4S1 


150 


103 


•50 


Math 


DiO 


28i ‘ 






103 


1 


Instructional TItt; 


1 


1 


1 


1,2'/ 


1.25 


(In hours) Kith 

Fad) it Ira 

/ 


I 


1 


* 


1.25 


1.25 


2 ‘ cla .siooms 


^ ti ai 1( T s 


4 trailers 


2 Sfl cet.ters' 


1 clastroon 


1 clanarooti 


Space J 

1 


’ c 1 ar.ari'oma 


1 cl toon 


1 dbl center^ 


1 activity area 


1 activity area 


— - 


900/1000 


icoo/)ooo 


; relnf oicctnent 








• lotal square feet 


S600 


4C0o 


. 8OC0 


2000 


2000 


2000 


Air conditioned 


X 


X 


X 


X 




X 


Carpeted 


X 


X 


X 


X 


X 


ft 


Special wiring 


X 


X 


X 


X 


X 


ft 


Carre) * 


X 


X 


X 


X 


r. 




Tablet 


X 


X 


X 


X 


X 


ft 


Utiliratlon 














Time in vfc ) (? hr)s!i If ta 


3(2-hr)shlfta 


7 per lot* a 


5 


5 


5 


Student /inf. 1 1 net ional unit 


20 


20 


40-S; IbA) 


50 


50 


75 

40 


Area/atudent (aq ft) 


bO 


50 


50 


40 


40 


S t a 1 1 i ti g 








1 




1 

1 ‘ 


)eatliet v/cn.U't vi uiilt 




\ 


1 


1 


Faraprof ('fcsU>t«.il s/iinlt 
Students per tfjct.*r 


1 

20 


20 


} 

40/60 


2 

50 

1 • 


j 

50 


25 

2 


Tecchers per piograa 


C 


5 


4 


1 


2 


Paraprofess lon-ilt/progr m 


6 


5 


5 


2 


3 




Other direct 
fquipiDcnt 












Cass'*tte 


Dorsett M'fib 


EDL AUD-X 


Hoffman readers 


Telex 


el ex 


( 


Teaching 


Control led 


Tape recorders 


Caasette 


Caaset te 


players 


Major ltcn.s \ 


r.^cliine 


readers 


Flashcard rdra 


recorders 


recorders 


Tape rrrorders 




Tach-X 


IVorg-Wai ner SO 


Tape recorders 


Tape recordcra 




\ 




riash-X 


(backup) 




Language a.aster 




Haterlala 

Prograo-reletcd 


1 

nirstr fps 


Fi Instr Ips 


Hof! Ran r^tls 


FRl. ruteriala 


BRL tnaterlala 


Fllrst rips 


Records 


D ! acs 


EUL materials 


C.iaset tea 


Caaaet tea 


Casnr t tes 




I'orsett 


EDL materials 


Great variety 


Variety of 


Variety of 


Papet backs 




taterials 




8org’Warner piatls other 


other 




Consunnblca 


1 












(student -related) 


» 


X 


X 


X 


X 


* 


Pre-aervicc trnlni »g 


1 










1 week 


Teachers 


2 weeks 


1 week 


2 weeks 


1 week 


I week 


Pa rspr of css Iona Is 


2 s'ceks 


1 week 




1 week 


1 week 




Other staff 


1 


— 


- — 


1 week 


3 daya 


3 days 


In^acrvlce ttailnlng 


S days 


— 


2 hr/wk 


4 daya 


Other Support 


1 


. 










Student diap(u’>utlc ser\'lces 


1 — 


— • 




x^ 


x^ 


— 


Program evuiunt ion 


1 X 


X 


X 


X 


X 


X 


Consultants 


6 days 


8 days 


8 daya 


6 days 


8 daya 


8 days 



*Two 75-^ulnutc periods for 1-4 with reinforcement In regular cltMiies. One 2.2i-hr period grudct 5 and 6. 

^Cach crntcr hob an ins true t lotul area pi ts an activity area. 



rtmotc dinguostlc and piehcrlptlve aerv!ccj». 
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Tlio (loll, It cor, I Infotnintlon for tlu’r.o 1 1 1 ufM rnt i vo pmr.iMiu*? Ifi nliown 
shown In Tnhlo 3. TIu'no nrc for Ihn oHllmatrs of tht* comp;ir,'ihl(* rcpll- 
cntlon cost. 3’hl«; InfomnUon Is cc nhlncd wllli the pror.r.im ,ind rcsourco 
Inforinnf lon of 2 nnd providon thr bnslc Input ItiformiTl Ion for thn 

pl.innlnp, coot mode*]. 

The fitnndnrd Input rofltr. ,Tud tlio factors for use In the plnnnlnp, 
coot inodol arc f;lvcn in Klj;, 10. The tenn "stnndnrd” is used no a de- 
scription of the factor used ncroos nil proHrmns. 

A cost of $12,000 per year per tcnc!u!r Is used In the model to 
cstlinnte the comparable rcplirntlon cost. This Includes tlic frlnp,c ben- 
efito (fixed charpes In most district ncrounlinp, systems). This is ob- 
viously out of line for, sny, n small rural district in the southeast- 
ern part of the country. But because tliio factor w,is used for all the 
proRr.ams, the different cost for the salary expense of the propram cost 
actually refleato the difference in the nwnber of teachers needed for 
the propram. 7’his same arpument applies to all the standard resource 
costs and factors used in the planninp cost model. 

The compa'ablc replication cost for each of the illustrative pro- 
grams is given in Table A. The acquisition cost includes the cost to 
‘remodel and furnish the instructional centers, the cost of the equip- 
ment a 'd the materials needed for all the instructional centers, and 
the pre-service training cost of the program staff. The operational 
cost includes the salaries of the staff, the cost of materials consumed 
or lost through attrition or theft, the cost of replacing and maintain- 
ing the equipment, the cost of in-service training, and other support, 
which includes a program evaluation cost on a per-student basis per year 
and consultants required during the year. The comparable replication 
cost along with the relevant dimensions of the specific programs is 
summarized in Table 5. 

The estimation of the comparable replication cost has an advantage 
in addition to adjusting for variations In the resource prices so that 
the cost of programs in different districts is on a comparable basis. 

This advantage lies in the discipline necessary to organize the program 
information and the cost information. In Table 5, the operational cost 
per student per subject offers a quick comparison of the relative merits 
of the programs. The other data of Table 5 can he analyzed In a simil.^r 
fashion. Care must bo taken, however, not to develop misleading 




•'results. •• 
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Tab ].c 3 

PROGRAM COST IN’FORmTION 
(Costs In dollars) 



Equipment Cost 


Frog ran A 


Progr.vn B Progrevn C 


Pragi'an D 


Frogro)n K 


Program F 


Total 


20,400 


15,000 


37,000 


2,500 


2,000 


5,000 


Cost per instructional area 


3,400 


3,000 


9,250“ 


2,500 


2,000 


2,500 


Number of instructional aieas 


6 


5 


Ia, 


I*’ 




2 


Stuvionts per instruct*! area 


20 


20 


40/65“ 


50 


50 


25 


Replacement — 10 percent 


2,040 


1,500 


3.700 


250 


200 


500 


Maintenance — 10/20 percent 


4,080 


3,000 


7,800 


250 


200 


500 


Materials Cost 


Total 


18,000 


>0,000 


t^5,000 


8,000 


8,600 


7,600 


Cost per instructional area 


3,000 


4,000 


n,250<'i 


8,000 


8,600 


3,800 


Number of instructional areas 


6 


c 


4 


2^ 


2 . 


2 


Consumables ($ per student) 


10 


. 10 


10 


10 


10 


■ 5 


Pro-scrvicc Training 


Numher of staff days“ 


120 


50 


90 


15 


2Q 


20 


Cost per day^ 


200 


200 


' 200 


200 


200 


200 


Total cost 


24,000 


10,000 


:.a,ooo 


3,000 


4.000 


4,000 


In-service Training 


Number of staff-days 


30 


— 


32 


12 


12 


12 


Cost per day 


200 


— 


200 


200 


200 


200 


Total cost 
Other Support 


6,000 


— — 


6,400 


2,400 


2,400 


2,400 


Student diagnostic services 


— 


— 


. — 


50^ 


50^ 




Student evaluation ($/studont) 


10 


10 


10 


10 


10 


10 


Consultants ($100/day) 


890 


800 


800 


800 


800 


800 


^Cost per center includes reinf orccuent 


areas. Single 


center cost 


slightly more than cost 


sf^own , 



One clnssroom area plus one activity area. 

c 

Forty students per sinp,le center, sixty-five per double. 



Includes time for raraprofessional training. 

Includes salary, materials, and training costs, 
f 

Remote diagnostic and prescriptive services,’ 
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Facilitics 

Remodel inp, (including carpeting, 

airconditioning, etc.) $ 3,000/center 

Furnishings (including carrels) $ 2,000/ccntcr 

Fquipmen t 

Replaccncnt 10% 

Maintenance (depends on estimate of 

1 .. liability based on complexity) 10% or 20% 

Materials 

Attrition from use, theft 10% 

Consumables $10/studcnt 

Salaries ^.including fringe benefits) 

Teachers $12, 000/year 

Parapro^essionals $ 5,000/year 

Specialists $12, 000/year 

Program directors $15, 000/year 

General support $10, 000/year 

General administrative $12, 000/year 

Consultants $100/day 

Pre- and In-service Training (Including 

salaries, materials, training) $200/day 

Program Evaluation $10/student 



Fig. 10 — Standard resource costs and factors 
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Table 4 



COMPARABLE REPLICATION COST FOR THE ILLUSTRATIVE PROGRAMS 

(In dollars) 



Acquisition Cost 


Prog ran A 


Program B 


Progveen C 


Program D 


Program E 


Progvm F 


Facilities (remodel, furnish) 
Total program cost 


30,000 


25,000 


2.0,000 


7,500 


7,500 


10,000 


(Cost/lns tructional area) 


(5,000) 


(5,000) 


(5,000) 


(3,750) 


(3,750) 


(5,000) 


Equipment 


Total program cost 


20,400 


15,000 


37,000 


2,500 


2,000 


5,000 


(Cost/instructional area) 


(3,400) 


(3,000) 


(9,250) 


(2,500) 


(2,000) 


(2,500) 


Materials 

Total program cost 


18,000 


20,000 


^■5,000 


8,000 


8,600 


7,600 


(Cost per instruction.^! area) 


(3,000) 


(4,000) 


(11,250) 


(8,000) 


(8,600) 


(3,800) 


Pre-service Training 


24,000 


10.000 


:.8,ooo 


3,000 


4,000 


4,000 


Total acquisition cost 


92,^'iOd 


70,000 


12'0,000‘ 


21,000 


22,100 


2 6, 'odd 


Operational Cost 

Salaries (incl fringe benefits) 
Teachers ($12 ,000/yr) 


72,000 


60,000 


48,000 


12,000 


12,000 


24,000 


Paraprofcssionals ($5,0C0/yr) 


30,000 


25,000 


25,000 


10,000 


15,000 


10,000 


Other (variable) 


— 


— 


— 




— 


— 


Materials 

Program-related (10%) 


1,800 


2,000 


4,500 


800 


860 


760 


Consumables (student) 


3,500 


2,850 


5,000 


1,500 


1,030 


2,500 


Equipment 


Replacement 


2,040 


1,500 


3,700 


250 


200 


500 


Maintenance 


4,080 


3,000 


7,800 


250 


200 


500 


In-scrvicc Training 


6,000 


— 


6,400 


2,400 


2.400 


2.400 


Other Support 


Student diagnostic services 


— 




— 


.7,500*' 


5,000^ 




Student evaluation (testing) 


3,500 


2,850 


5,000 


1,500 


1,000 


2,500 


Consultants ($100/day) 


800 


800 


800 


800 


800 


800 


Total operational cost 


123,720 


98,000 


106,200 


37,000 


38,490 


43,96‘0 



^Remote diagnostic and prescriptive services. 




Table 
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For exrimplc, the ncf|ul sition cost per studcMit could be obtniiUH!, 
it seems, simply by dividing the arq\iisition cost bv the number of stu- 
dents. The problem lies In just wliat 'bmml)er” of students to use. If 
the total number of students in all tlie insti'urtional periods (or some 
such time cllvi£;lon) is used, the acqviisition cost per student reflects 
an implicit utilization rate for ihe i nstrvictional center. A case in 
point is Program C, that program, the instnictiontal centers are used 

seven periods (or hours) each day. In current practice, that is the 
maximum utilization rate for facilities in any one. day. In Program A, 
on the other hand, if the instructional cenf.ers had been used for seven 
periods instead of six, one less instructional center would have had 
to be furnished. 

If the number of students per instructional center is assumed as 
"best,** then the acquisition cost on a per-student basis for each in- 
structional center for each program can be obtained and qualified by a 
statement of the utilization rate of the instructional centers. An ob- 
stacle is encountered in using the acquisition cost per student per pro- 
gram. That is, that the equipment and materials purchased for one year 
will have more than one year's service as the program is continued. In 
short, the use of the acquisition cost per student as an indicator of 
program cost is fraught with hazards. These hazards are explored in the 
section on estimating the incremental cost of a specific program in a 
particular district. 

Estimating the Incremental Cost 

The comparable replication cost serves as an **lndex" cost for use 
in the comparative analysis of different programs . It does not answer 
the question of what a new program might cost if implemented in a spe- 
cific school district. The incremental cost to the district is necessary 
In making decisions about whether or not the district can afford a pro- 
gram similar to the successful program in another district. This cost 
is necessary when deciding the scope and the design of the program that 
can be accommodated within the resource constraints of the district. 

The process of estimating the incremental cost is essentially tlic 
same as the process of estimating the comparable replication cost. The 
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cmphnsis is on estimating, the resource requirements and on translalinp, 
these requirements into an estimate of cost. In some districts, the un- 
availability of certain resources ini(jht be an obstacle to the implemen- 
tation of a program even though the district had t!ie funds to afford the 
program in an accounting sense. This possibility makes it all the more 
Important to estimate the physical resources needed to implement and op- 
erate a program. 

In estimating the incremental resource requirements, the resources 
available within the district at no additional cost are taken into account. 
These resources could be, for example, assets Inherited from discontinued 
programs, pliysical resources provided cost-free by the community, or vol- 
unteer services. After the net incremental resource requirements are de- 
termined, district-specific resource prices and cost factors are used to 
develop the estiirated incremental program cost, using the methodology of 
, the planning cost model. Specifically, the standard resource costs and 
factors shown in Fig. 10 are changed to district-specific costs. 

To illustrate the process and considerations in estimating the incre- 
mental cost of a program, the data for Program E (shown in estimating the 
comparable replication cost) will be used. These data are shown in Tables 
.6, 7, and 8. 

Data about Program E could have been generated by either the district 
of original implementation or by a state or federal agency in their evalu- 
ation of programs funded through the agency. Whatever the source, program 
data of this nature is essential information to another district in its as- 
sessment of potentially effective *'nev;** programs. 

In this illustration, it is assumed that information about all the pro- 
grams, A through F, was available and that Program E was tentatively se- 
lected as the most-likely-to-succeed program. Preliminary examination of 
the data used to develop the comparable replication cost (CRC) for Program E 
leads the district planners to belie^^e that the incremental cost to its 
district will be significantly lower. The"' district *s current salary sched- 
ule sets average teacher s«nlary at $9000 and paraprofessionals at $4000. A 
major portion of the equipment and materials required for the program are 
available within the district. 



O 
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Table 6 

PKOGI^M AND RlilSOURCn INFOl^TTON FOR PR0(;RAM E 
An Elementary Level, Reading and Mathematics Program 



Desoi'^i^ptArpQ 
Students Served 



Instruction 
Class time 

Number of students 
Students /instructional area 
Number of sections 
Utilization 
Facilities 
Space 

Furnichings 

Staffing 

Certified teachers 
Special teachers 
Paraprofessionals 

Equipment 



Materials* 
Pre-service Training 
In-service Training 
Other support 



Pcsourae Information 

Grades 2-4 
Title I; low SES 
Underachievers 

1.25 hours - Reading 
1.25 hours - Mathematics 
103 
50± 

2 

5 hours /day 

2000 square feet 
1 instructional area 
1 activity area 

6 carrels 
Carpeting 
Tables and chairs 

1 per instructional area 
None 

2 per instructional area 
1 per activity area 

Telex (remote diagnostic) 
Tape recorders 
Cassette players 
Headsets 

Books, games, incentives 
5 days - formal 

3 days - formal 

Remote diagnostic-Prescrip- 
tive services 



Quantity and quality of items would be specified in 
supporting lists. 



1.A 
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Tub le 7 

COST INFOICtATlON FOR PROGRAM E 
(Costs in dollars) 



Item 



COGt 



Facilities Cost 
Total prof^Tam cost 
Cost per instructional area 
Equipnent Cost 
Total 

Cost per instructional area 
Nuinl)cr of instructional areas 
Students per instructional area 
Replacement factor 10% 

Maintenance factor 10% 

Materials Cost 
Total 

Cost per instructional area 
Numl^er of instructional areas 
Consumables ($ per student) 
Pre-service Training ^ 

Number of staff days 
Cost per day^ 

Total cost 
In-se\*vice Training 
Numljer of staff days 
Cost per day 
Total cost 
Other support 

Student diagnost^’ c services 
Program evaluation ($ per student) 
Consultants ($100 per day) 



7,500^ 

5.000 

2.000 
2,000 

50 

200 

200 

8,600 

8,600 

10 

20 
200 
A, 000 



12 

200 

2,A00 



10 

800 



One instructional plus one activity area. 

Includes time for paraprofessional staff, 
c 

Includes salary, materials, and training costs. 
Contracted diagnostic and prescriptive services 



1 
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Tab] c 8 

COMPARAlibK RKPLICATION COST FOR PROGRAM E 
(In dollars) 



Item Cost 

A cfH3lsitlon Cost 

Facilities (remodel/f urnish) 

Total program cost 7,500 

(Cost per instructional area) (3,750) 

Equipment 

Total program cost 2,000 

(Cost per instructional area) (2,000) 

Materials 

Total program cost 8,600 

(Cost per instructional area) (8.600) 

Pre-service training A ,000 

Total acquisition cost 22,100 

Operat ional Cost 

Salaries (including fringe benefits) 

Teachers ($12 ,000/year) 12,000 

Paraprof essionals ($5 ,000/year) 15,000 

Other (variable) 

Materials 

Program-related (10%) 860 

Consumables (student-related) 1,030 

Equipment 

Replacement (10%) 200 

Maintenance (10%) 200 

In-service training 2,A00 

Other support 

Student diagnostic services 5,000 

Program evaluation 1,000 

Consultants 800 

Total operational cost 38,490 



Diagnostic and prescriptive services by 
contracted services . 




1 
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For this district, tlio CRC for Prop.r.im li represents a maximum ex- 
pected prop^ram cost. For another district, with a hiplier salary sched- 
ule and no equipm(!nt or materials on hand, tlie CRC for .Program E would 
be lower than its incremental cost. Both districts Rain needed insiplits 
about the cost impact of Program E from Just a quick look at the CRC 
/ for Program K. These Insights cannot be developed if the only cost in- 

formation the district has about Program E is a cost per student or 
the total program cost specific to the district originally developing 
the program. 

In developing the program cost estimates for use in designing the 
scope and nature of Program E, the district determines the resources 
available within its Inventory and matches this information with the 
resources requl..ed to Implement and operate the program. The result- 
ing incremental resource requirements are translated by means of the 
• planning cost model into an estimate of incremental cost. In this* 
translation process, district-specific resource prices and factors are 
used . 

The data needed and the results of the incremental cost analysis 
for the various configurations of Program E are presented in the same 
formats as Tables 6, 7, and 8. As an Illustration, the incremental 
cost for two program configurations (160 students and 200 students) is 
shown in Table 9. The assumptions. Incremental resource requirements 
and district-specific resource prices supporting the cost estimates 
would be displayed, in practice^ in the formats of Tables 6 and 7. In 
this illustration, most of the Information can be identified in Table 9. 
Just briefly, the district has in inventory about 50 percent of the re- 
quired equipment for a program of 100 students. Adequately remodeled 
space is available for one instructional area and one activity area. 

But, two instructional areas and activity areas are needed for 160 stu- 
dents. Only carrels have to be purchased in order to furnish as many 
as four centers. For one configuration, the district looks at the cost 
Impact of developing an in-house capability for the diagnostic-prescrip- 
tive services that are provided to the other configurations on a con- 
tracted basis. This lends to an increase In the cost of pre-service 
training and the additional operational cost for staff members to pro- 
vide this program-related service. 
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Tab] c 9 




f 



IHCUr:MFNTAL COST i:STIM/\Ti:S FOR ALTFRNATIVF CONFIGUJ^MIONS OF PROGRAM E 



(In dollars) 



Progrcvn Coot Category 

Acquisition Cost 

Facilities (Remodel/ furnish) 

(1 instructional and 1 activity area 
have to be remodeled) 

Equipment 

(Unit cost/instructional area for AO 
students is $ 2 , 000 ) 

Materials 

(Unit cost for instructional area 
for AO students if $6,500) . 
Pre-service Training 

(5 days per staff member and 
training of forty days for diag- 
nostic services in E 3 

Total Acquisition Cost 

Operational Cost 
Salaries 

Teachers ($9,000) 

Paraprof essionals ($A ,000) 

Other ($5,000/1/3 time) 

Materials 

Program- related 
Consumables 
Equipment 
Replacement 
Maintenance 
In-service Training 
Other support 

Student diagnostic services 
Program evaluation 
Consultants 

Total Operational Cost 



160 a l^udants 200 otudentr, 160 students 



3,500 


3,500 


3,500 


3,000 


3,800 


3,000 


13,000 


17,200 


13,000 


4.000 


8,000 


12,000 


23,500 


32,500 


31,500 



(2) 


18,000 


(2) 


18,000 


(2) 


18,000 


(2) 


8,000 


(6) 


24,000 


(2) 


8,000 




1,300 




1,720 




1,300 




1,600 




2,000 




1,600 




400 




500 




400 




400 




500 




400 




3,200 




6,400 




3,200 




8,000 




10,000 




— 




1,600 




;>,ooo 




3,200 




800 




800 




800 




43,300 




65,920 




41,900 




1 
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Tlie rciSulUnf' pro);rnm coat analysis provlHos tlic Information needed 
by the district In making; the decision about whether to plan the iuiplc- 
mentation of the prop, ram and, if so, what configuration of propram can 
be afforded within the resource constraints of the district. As a final 
note, two points should be made clear. First, these cost estimates 
are planning cost estimates. Much greater detail and accuracy are re- 
quired to meet the needs of actual implementation and financial accoun- 
tability. Second, analysis of the dollar-cost alone does not provide 
adequate information for educational decisions; for this reason the 
emphasis here is on the analysis of both the dollar and non-dollar re- 
sources required for alternative programs. 
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Appendix 

DETAILS OF PROGRAM AND RESOURCE INFORMATION 
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Table 10 



PROGRAM AND RKSODRCE INFORMATION 



DcGcrtpiov3 



Progi'am A 



Information 



CJiaracterisLics of Stuilcnls Grader, 7-12 

Served Educationally disadvantaged (at least 2 

years below level) 



Number of Students 



350jReading 

math 



Instructional 

Class time 

Class sixe 

Facilities 

Space 

Students/ class room/ day 

Utilization 

Furnishings 

Staffing 

Teachers 

Special teachers 
Paraprofcssionals 
Other personnel 



fl period Math 
^1 period Reading 
20 students per classroom area 

4 trailers 0 900 sq ft 
2 classrooms 0 IGOO sq ft 
6 hr/day; three 2-hr shifts 
Desks, carrels, carpet, air conditioning 

6 

0 

6 

Project manager; associate manager 



Equipment 

Materials 

Pre-service Training 
In-service Training 



Dorsett M-86 Teaching Machines 
Filmstrips, records 
1 week per teacher 
5 days total 



Other Support 



Incentives 




PROCR/J-1 AN'D RI'SOURCi: INFORflATION 





Program b 


' Dcacripior 


hifortnatioyi 


Chaiacto.rlst ics of Students 
Served 


7-12 grades 

Educationally handicapped (at least 2 years 
below grade level ) 


Number of Students 


285 (Reading 
\Math 


Instruction | 


fl period Math 
1 period Reading 
[20 students per classroom area 


Facilities 

Space 

Number of students 

Utilization 

Furnishings 


A trailers @ 900 sq ft 

1 classroom @ 1000 sq ft 

20 per classroom area 

6 hr/day; three 2-hr shifts 

Desks, carrels, carpeting, air conditioning 


Staffing 
Teachers 
Specialists 
Paraprof esslonals 
Other staff 


5 

0 

5 

Project manager; associate manager 


Equipment 


EDL, AUD-X, Tach-X, controlled readers, 
Flash-X 


Materials 


Filmstrips , discs 


Pre-service Training 


AO hr per teacher and aide 


In-service Training 


No formal training 


Other Support 


None 
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Table 12 

PROGRAM AND RESOURCE INFORMATION 
Program C 



Deocriptora 



Characteristics of Students 



Instruct lonal 
Nunher of students (as of 
eild-Ueccr.bcr) 

Class tine 
Class size 

Nimber of sections 



Xnforvijtion 

Grades 6-9 

1 ?P0 

Transient rjQQ yearly turnover 

hlack, nodel cities ncighbortiood 
Low inCor.c 

Lowest achievers accord/np, to last spring’s testing 
Specialized pupils included 

Program pupils distributed atr.ong all homerooms 



A91 (Reading); 535 (Math) (sa:cc students) 

45 nlnutcs/day (Reading and Math each) 

35-40 in single center (SC) (40 optlmua); 60-65 in double 
center (DC) (optimum) 

14 each (7-period day) 



Facilities 

Sp«3ce 



Students/ classroom/day 



fA centers: 1 DC for reading and math; 1 SC for reading 

and 1 SC for math; each center has an Instructional and 
an AM^ arfa 
1 relnf orcerent room 

^total occupies space of 7 foruer classrooms (walls were changed) 

N o, s tudents per day • ( 491 535) ^ 

No. classrooms 7 



Furnishings 



Staffing 

Certified teachers 
Special teachers 

Parapr ''esslonal 
Otlier personnel 

Equipment 

Primary unit 
Supplementary system 

Redundant system 



Ititerln^ I (102! consumable) 



^Tablc space for carrels 
Carpet Ing 
< Air conditioning 

1 carrel per student per class (l.c.» approximately 140 total) 
Xhairs 



1 per center (Reading and liath each) 

None 

[Full-tlroe: 1/centcr; 1 for reinforcement room 

[1 Substitute 
fl full-time director 
[1 full-time secretary 



Ucadirtg 

40 Hoffman Reading machines 
25 tape recorders/center 
(50 total) 

* 15 Borg-Warner 



Math 

40 tape recoi .;crs/ccnter (80 total) 
40 flashcard leaders (Electronic 
Futures, mfg.) 

System 80 — » 



^2 sets EPL tapes/center Math mini system (tapes) 

2 sets Hoffman materials Workbooks (not on per pupil basis) 

(levels b to G)/center 
'workbooks (not on per pupil 
^ basis) 



2 sets Borg-Warner materials (levels 1-6) per reading and math 
center (i.e., of complete sets) 

1 notebook per student for compiling materials 



Prc-cervlce training 



{ One week on AMS in-depth training 
One week going through materials 



Jn-servlce training 



About 2 hr/wuek 



Other Support 



NonC| Instructional program self-contained 



1 : 






-38- 



Table 13 

PROGRAM AND RESOURCE INFOR'^IATION 
Program D 



Descriptors 



Information 



Characteristics of Students 



{ Grades 1-6 

Inner-city , black , low income 
Transiency = 30% 

Lowest achievers for first 5 months, then 
entire school (excluding most special 
, education students) 



Program Scope 
Instruction 
Number of students 
Class time 

Class size 
Number of sections 

Facilities 

Space 

Students/ classroom/day 



Furnishings 



Staffing 

Certified teachers 
Special teachers 
Paraprofessionals 
Other personnel 

Equipment 

Telex 

Cassette tape records 



Reading and math 

Initially 100, later 150 (as of February) 
Initially 2-1/4 hr, later reduced to 75 
minutes for grades 1-4 
45-55 (maximum at 60) 

Three (one each for grades 1 and 4, 2 and 
3, and 5 and 6) 

Two regular classrooms 
75 

( 30 carrels and chairs, with electric out- 
lets at each carrel 
7 tables , 21 chai rs 
13 bookshelf-cabinets 
VCarpeting 



One (no outside preparation required) 
None 

Two, 6-hr day 

On-site director and secretary 

1 

30 



Materials 

Pre-service Training 
In-service Training 



BRL modern math texts 

Large variety of other materials 

Five days for entire staff of school 

Eight morning meetings for entire staff 



Other Support 



None 
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Table lA 

PROGRiVl AND RESOURCE INFORMATION 
Program E 



Descriptors 

Characteristics of Students 
Served 

Instruction 
Class time 

Number of students 
Class size 
Number of sections 
Utilization 

Facilities 

Space 



Furnishings 



Staffing 

Certified teachers 
Special teachers 
Paraprof essionals 

Other personnel 
Equipment 



Materials 

Pre-service Training 
In-service Training 
Other Support 
Incentives 



Information 

Grades 2-4 ; Title I 
Low SES 

1.25 Reading 
1.25 Math 
103 

50 studencs per class 
2 

5 hr. per day 

'2000 sq ft 
^ 1 classroom 
1 activity area 

'6 carrels 
< Carpeting 
Tables 

1 per center 
none 

12 per center 
\1 per activity area 

I Telex 

Tape recorders 
Cassette players 
Headset 

Books, games, toys 
5 days 

4 days, total 

Remote diagnostic and prescriptive 
25 per student — candy, scrip 













Table 15 

PROGRy\M AND RESOURCH INFOR^IATION 
Program F 

Dee crip tors Informati on 

Characteristics of Students 

Served Title I students 

Number of Students 250 

Instructional 

Class time 50 minutes 

Class size 25 

Number of sections, school 5 . 



Facilities 

Space 

Students /classroom/day 

Utilization 

Furnishings 



Staffing 

Teachers 

Special teachers 
Paraprof essionals 
Other personnel 

Equipment 



Materials 



Pre-service Training 
In-service Training 
Other Support 
Incentives 







Regular classrooms 
125 
100 % 

Air conditioning, pleasant environment 
small, modern (partitions, file cab- 
inets, storage cabinets, etc., loose 
table , chairs) 

1 classroom 
0 
1 

1 program director 

6 Cassette players ($25) 

6 tape recorder ($150) 

Earphones ($50) 

Sound filmstrip sets 
Cassettes 

Workbooks and miscellaneous supplies 
1 week 
3 days 

Evaluation: $10 per child 

300 books given as awards 



h 
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PERFORMANCE TOWARD WHAT RESULT; 

AN EXAMINATION OF SOME PROBLEMS IN OUTCOME MEASUREMENT 

Selma J. Mushkin 
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ri:m’C)]iM;u\ci: tov.'ard v/hat result: 



AN D.'AMIK'ATTON OF SOME FROELEiMS 



IN OUTCOME MEASUREMENT 



SeJjiin J, M'ashkin 
Public &Oi''vlcc;rj Lcboi-ctory 
Ge or jiet’ oivn UnJ.ver s it y 



Prepared for the 

National Conference on Perforinence Contracting 

December 9-10, 1971 

Belmont House, 

Elkridge, Maryland 



/ 

The research on v.'hich this paper draws was carried out 
under contract to the Eliblic Sei’vices Laboratory from the 
National Center for Educational Statistics. (OEC-0-70-4454 (521) 
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l’L.TO.■OU^fANC^J TOt/ARD WIAT RRSin/P: 
i\N EXAMINATION OF GOME PROBLK-IS IN OUTCOME ME/iSURE-IEi^T 



ScliDa J. Miif;hkln 
Georf’otovjn Univcroit:/ 



/ 



Up to a very short time ajro, the single criterion nehievemont of 
skills, such ns reading, x?ns applied in judging succosa or failure 
(or its score value) of nn education program. Tiie ioaue of a multi" 
dimensional educational product \7ns a subject of much debate; it was 
debated whether attitudea and attributes v;erc at all useful in 
assessing product, let alone necessary. Among the influences at work 
was the simplistic drive to get at "real measurements" of cost effective- 
ness in place of vague generalities about goals. Despite the merit of 
the undertaking, some basic concepts of effectiveness measurement v;ere 
set aside in the drive to shov; analytical studies would provide useful 
policy tools -- and to count program effectiveness. Discussions in 
1966 with OEO officials and with those in the Office of Education made 
plain this insistence that achievement tost scores, and those alone, 
served as sufficient measurements, hliatever else education might 
produce, was the argument, it surely must produce basic skills, 
such as reading, and if other products were of any importance, perhaps 
alternative programs or activities could achieve them better. To 
counterbalance this emphasis on achievement measurement was to question 
the whole exercise -- to question the .value of hard analysis. 
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KowaJnya in tlic coinrsc of nnalysir. of educational prosrainc and 

r 

thcii" evaluation, dimenniona otlicr tlinn achievement acores are in the 

ascent and it focnia hardly likely tliat present trends v.’ill be reversed. 

Partly this is a consequence of the findings of tlio evaluation studico 

themselves; partly it is a result of more interdisciplinary effort in 

which the skill?; of philo.'.ophy , sociology, psychology, and anthropology 

arc being brought to bear on educational outcomes along v;ith the tools 

of tlie econoini.st. In the last five years or so, educational research 

has shomi cons:!. s ton tl>' that various characteristics of the children's 

.^ome enviroruiicnt, jparticulnrly parental attitudes strongly influence 

(1 2 ) 

children's achievement, ’ Parental aspirations for the child have 

been found to be batter predictors of school attainment than variables 

related to the school itself. At the same time, research on the 

development of intelligence and achievement has indicated the gains 

that come about from experience, especially social experiences, ero 

(3 4) 

essential ingro.dicnts in the developmental process, ' ' Teacher 

expectation as part of that social cxpoi*icnce has been shoxm to 
influence learning. 



Basic criteria of effectiveness for evaluation purposes call for a 
series of properties in the criteria used. These include (1) relevance 
to the objectives being measured; (2) completeness in encompassing the 
whole of the objectives fonmulated; and (3) measurability. 





■ Jc 



While use of Inaufflclcnt moneurementa or niouciireinonta that tnatch 

inadequately t!io full range: end meaning of the objcctivoa formulntecl 

rccuiirccl caution in acceptance of ros\ilta of evaluation otudlca bacod on 

achievement scorca alone, the persistence of negative flndingu have had 

marked policy impact. On the basis of n review of ctudieo, one 

American erpert euggested "The difficulty seems to be that It is becoming 

clear that factors that can be varied \;i th money arc not vorj" closely 

( 5 ) 

related to die' achievement of school pupils," ' ’ 

< 

Feedback affects of education evaluation studies are reflected in 
the President's April 1970 Message to the Congress: "\^e will ask the 

Congress to supply many more dollars for education. , .but only if we get 
more oducatlor for the dollar," Hie President led the nation in saying 
that the most pressing need of the schools is not for more money, but 
for reform. The aucstioning of state aid for schools and the enlarged 
opposition to property tax Increases for schools follow from the 
repetitive negative evaluation study findings on achievement scores. 

Emphasis on achievement scores haa had an Impact on processes of 
education as v;ell. If the achieving of skills or proficiency in such 
matters as reading are, in fact, the purpose of the school, it is but 
a email step to: 
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,,, Tlic undcrtnking to tencli to the specified skills thnt nro 

rcfpiircd, (Vnion the skills soiij 2 ht are rlRoroiisly tdontif led 
and r.peoificd, and niansnreo of those skills are designed and 
Implenientod , tcnclilnp; progrniriR can be diiroctcd. If the oklllo 
learned are deficient , a special program con bo designed, c.g., 
right to rend program.) 

,,, Tile review of incentives for the teaching of those specified 
akills, (If the skills arc knoi'/n and methods of teaching os 
v/ell, then presumably the implementation of the method can bo 
encouraged by Incentives.) 

... TIic design of new incentives for performance of teacher and 
student, processes for. encouraging teaching and 

learning are not readily at hand, then additional research 
on incentives should prove useful.) 

, New ejcperlncntotlon v;ith teaching methods v?hen processes 
are not known. 



Performance contracting has its origin in this emphanls on achlovcu'.. it 
scores. It provides both a means to stimulnto learning and teaching of 
the specific skills, and also a base for nevj c 3 :perimentatlon V7xth 
processes of teaching and learning. 



O 
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In the current scene, performance of education tOT/ard educational 

r 

results has major importance, Tlie difficulties of defining results sought 
or outcomes, and of assessing by measurements those results and outcomes, 
are many. Among the steps required are these; 

1, To define outcome appropriately in terms of purpose, and if 
necessary, to define outcome inultldimenslonally . 

2, To explore what is knovm about measurement of the defined 
outcomes. . 

3, To examine the state of the art on Instrumentation to improve 
the measurements, 

A, To assess interaction and multico llinearlty of the defined 
outcomes. 

5, To select proxy measurements for use, 

6, To test and retest through use the outcome proxies selected, 

7, To Identify from among the tooting Inatrumeuto those that 
could be applied aa proxies. 



(V.v 
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SLcpa ttikon to cloljino. oiiLcomo wp.n.‘;uromc;nto end to implcnunit cuch 

measLirciuontn Cfivry additional reuponoibilitlQr) for tho fcedbaclc effectn 

arc many nnd can hardly be icnored, Among tho feedback effccto arc theoo: 

... Concentration on the "proxy" rather then tho underlying 
learning nought, 

... Direction of fundr, to improve the "pro:y," for exomplo, 
the roadifjg rordinc;i!.i program, 

... Poosiblo changes in tho learning of children. 

If reading skills ‘of children are rewarded, then reading acorea 

« 

tend to bo emphasized both in teaching and resource allocation, A numbern 
game is generated on the pro?y. Proxies intended as simplifiers become popu- 
larizers, or stated differently, tho focua ia on the prox>' measure end 
data collected about it. Ue noted tho right to read exeraple given earlier. 

It is simply symbolic of a range of feedback -- that ia from pro;y to 
policy. There arc other feedback effects aa well. The projy rccasurementa 
as is indicated later, can affect the child himself and hia learning. 

Among the additional stops that need to be taken are these: 

1, To assemble what Is known about the impact of measurable 
factors on learning, 

•.i 

2, To study feedback effects on learning of criteria for 
judging learning, 

3, To study feedback effects on policy of criteria for learning, 

A, To determine v?hat Is Icnown about how to create each of 
the Identified outcomes. 



In the preaent paper I have tried to consider oome of the problcinn of 
mcasiiremcntc , vlowlric mear.urcmcnta from Lho percpective of performance 
as con5>uinption (both in the inimcdicte period and in the longer run), and 
also os r.n investment, Tlie research on vhlch thlo paper draws vraa done for 
the l.ational Center for Kducational Statiatlco In a small ocalo exploratory 
utudy of statistical nienr^urcmenta of education/.:! oiitcornec. In that rtnr.ll 
initial project, only a bep/inninp, could bo made on the many problems 
involved in outcome monsureinent. By and larpc the emphacis has been on 
a taxonomy of outepme measurements and review of data sourcen and 

I 

instrumentation that could be applied in a ototistical gather g effort. 

Thus "pad and pencil" tents are those that ore given primary attention. 

Outcomes of education appear to bo fou"-diracnaional , if on economist 
may be forgiven for intruding a description dealing v;ith economic 
imraeasurables. A review of some of the worl: that has been done on education 
outcomes in other disciplines suggeato that these outcomes may be classified 
as the four A's: (1) attributes; (2) aptitudes; (3) attitudes; and 
(4) achievements. Together the proxies chosen for these four A's would 
enable the exploration of a number of multisectoral aspects of education 
and perhaps provide a better understanding of the dynamics of educational 
outcome. With the four general sets of criteria, it \;ould appear possible 
to ^robc much more- deeply into the results of education and to ascertain 
whether earlier evaluation otudics ore wide of the mark or, in fact, ore 
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sufficient. 
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Complcxltlcf! fire even j;j*e«ter, for eJncntlon hna both uhort-run 
procUici;!; and long-term onea, bliat product definition coptureu each 
of tlic pliacca In a ccntinuiim that crcrtcu the "being" of boy, and then man? 
Charnctcrictlcf.lly education in economic terma may be rogorded both no on 
Invcatmcnt and ok n conaumer good. Outcome deflnltlono oosociated with 
each of thoac purporcc are very different and the conacquenceo of the 
uiensurea for policy differ, too, 'llie "invcat’iient" may be tested in 
terms of rat.eo of return, or by changeo in employment, occupations, and 
earning''. As t conaumer good, education la perhaps deccrlbed more vaguely 
as to quality and quantity of product. Education as o conaumer good 
gives "v;oll-being," "joy," "happineos," In one aapect the conaumer good 
is "durable"; in others enjoyment may be momentary. In common with other 
market goods, ?t may bo tooted by consumer decisions taken. Votes on 
school bond iatucs provide a measure, but even such decisions arc subjci.' 
to much ambiguity and not unimportantly, even the decision about who 
is the consumer Is not clear -- the child or his parent? Public school 
systems operating as they do, an monopolies, "mean consumer cholcca are 
na,. rowed greatly. Voting behavior rather than market behavior records 
the public choice, 

1, Short-run and Consumer Mensuremeuts 



changes in behavior that come about through changes in attitudes, ferreting 
out and development of aptitudes and attributes, and the nchlovcracnt of 
bkllls. Outcome measurements can hardly Ignore behavior chango so central to 
the objectives of educational Eervlcco, 



Essentially product definition in education concerns Itself with 
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Acliicvcmcnt nncl aptitude (or ability) tenting ao "product" indexea 
have received much of tlie attention in the patJt among the "soft" oocial 
Gclencec; economialG only recently directed their attention to ohort-nin 
resultc, Jeremo Bruner summed the concern \?lth meaohrlng nchievemento in 
this way: Schools do not look to the creation of nelf-conf ident fools, 



A ptitude "••• Aptitude meaourementc arc perhaps the earliest of the 

measurement undertakings and the most widely applied. Aptitude (defined 

here to be synonomoii s with ability) is not to be regarded, the social 

sciences' findings -show, as an inherited "static" characteristic. Ability 

indexes have been used by economists as if they were independent of 

(7) 

"educationl'; this use is not warranted by the research. Certainly 
ability as measured does not constitute a valid "correction" of school 
outputs to be introduced into production function or economic growth 
analysis. Part of the outcome in performance contracting ao in other 

educational processes is to discover and develop talents and to raise 

/ 



aptiti jde (abilit 7 /) levels. 



Achievements -- Following on the work of Alfred Binet and 
Theodore Simon, the idea of scale was applied not only to intelligence 
but to achievement testing as v.’ell, lliorndike and his students for 
example developed scales for measuring achievement in arithmetic (1908) , 
handwriting (1910) ,• spelling (1913), drawing (1913), reading (191A) , and 
language ability (1916), Other universitieo, particularly Chicago, 



\b^ 

ERIC , 



9 



joJ-UGcl ColumbiQ In Iti; ofro):t.o. And by 1918 tlui Nationnl SoclcLy for the 
Study of Eduention published itfj yearbook on llie MonRuremGnt of Educational 
Products. 

Tliorndike's \;ork in 1918 sounda much the oomo as the vocabulary of 
educational program analysts today, "Education is concerned with clicngcs 
in human beings; a change is a difference between tvo conditions; each of 
these conditions is knovm to us mainly by the products produced by It 
things made, words spoken, acta performed, and the like. To measure ony 

i 

of these products meana to define itc amount in aome way eo that competent: 

persons will know hov; largo it is, better than they would without 
(9) 

measurement," ' ^ 

Measurerao.it work on achievement and ability has continued at 
accelerating pree, spurred on in the inriiedlate period by the more 
widespread support for evaluation of educational programs. The Mental 
Measurements Yearbook was published in 1937; by the latest, or sixth 
yearbook published in 1965, 1,219 tests v^erc included, Iheir distribution 
by classification suggests that about 15 percent of the tests related to 
personality, a little over 11 percent to intelligence, and the remainder 
to achievements of such skills as English, mathematics, foreign langungea. 

Drawing on the. information provided in' surveys of state to.stin.r. 
surveys made by the Educational Testing Service in 1967, the Akron DljjIIc 
S chool Survey (April 1968) of basic testing programs used in major school 
systems, the 1969 Survey of Compensatory Education, and the Rcnchrcn 
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Council on Grootcr City Schools 1970 Survey -- It was found that 
some five or sl>: tests accounted for a large fraction of all the tcoto given, 
Tlie tests v/ldely used ore these: Iowa Tests of Educational Development, 

Stanford Achievement Tests, Sequential Teats of Education Progress, 

California Achievement Testa, Iowa Tests of Baoic Skllle, Metropolitan 
Achievement Teats, and SUA Achievement Series, 

For most tests given, notional nonns and regional norms have been 
developed by test publishers; and in those states in which state testing 
has been done for some period -- Nct; York, Alabama, California, Rliode 
Island, Minnesota, Pennsylvania -- statewide norms ore available, 

/ 

Measurements using achievement tests ns a proxy for the product 

"learning" are at best ' partial , and portion out, by "average group 

( 10 ) 

performance," learning into grade levels, "Tlaere la no wide agreement 
on specific subject matter for each grade,' !Hiere has been a great shift 
in what is considered proper learning for given ages and grades in some 
subjects, and it appears that even greater changes are in the offing." 

r- 

. Despite concerns about the meaning of a standard grade level - 
scores according to grade levels, particularly in reading, are widely 
applied, "Grade equivalent" simply means median performance in the 
normiiig sample at a given grade at the time of test standardization, 

Tlie norraing of the test and the representativeness of those norms 
clearly affect what is counted as grade level, 

1S8 
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Rof;cr Lennon's discussion of norms In a 1963 KTS paper notes: 

ILere are [-ood reasons for supposing that difforences In norms 
ascribahlo simply to, , .variations In norming procedures ard not 
negligible, V.lien wo consider that to such differences from test 
to test there must be added differences associated with varying 
content, (.nd \;ith tlic time at which r,tandnrdl. 2 ation programs are 
conducted (including the time of the school year), the Issue of 
compariibillty , or lack of It, among the results of the various 
tests may begin to be seen in proper perapeetive. Empirical data 
reveal that there may be variations of as much as a year and a 
half in grade equivalent among the results yielded by various 
achievement tents; varie.tions of as nuch as 8 to 10 points of IQ 
among various intelligcaice testa are, of course, by no means 
uuoCirmon, (12) 



The Office of Education has unden;ny an ANCHOR Test Study of 

/ 

achievement test batteries to develop score correspondence from rending 
test to reading tost. The development of such score correspondence is 
essential to any nationwide data collection effort that leaves to the 

r 

local couuuunit;'^ end state the initial decision on what children should 
learn and are learning, 

Iti that conte::t a feasibility survey v^as launched in 1969 on reading 
comprehension subtests for the most widely used standardized test 
batteries, appropriate for children in grades four, five, and sIk, The 
reading comprehension subtest of the five most widely used test batteries 
were administered to over 830 children, v;ith each child completing subtests 
from three batteries, arranged In a random order. From the test results 
computation was made of correlation coefficients among the five tests 
with the finding of a high correlation among tests. The correlation coefficient 
for groups of grade four pupils that was lovjcst was 0,81 and the highest 
correlation for this grade was 0,91, 





Based on the results of thl? feasibility survey, a major test 
equating and standardisation study is now undon^Joy, Among the purpooca 
arc these: 

, to set up nationally representative norms for reading 
comprehension and vocabulary subtests; 

,,, to develop tables of score correspondence between subtests 
of the Hetropolitan and corresponding subtests of levels 
of six other test batteries based on a ner? representative 
sample of population of children enrolled in grades four, 
five, and six in both public and nonpublic schools; 

,,, to correlate correspondence of- the several test batteries 
based on the newly collected scores from a sample of the 
appropriate national populations; and 

... to prepare tables of score correspondence for samples of 
several national subpopulations. 

Attributes and Attitude -- Ulille cchle’*ement scores received early 
use in program evaluation, attributes and attitudes are increasingly 
emphasized as part of educational outcomes. Tlie President in his April 
1970 Message to the Congress regarding education emphasized such character- 
istics as "humanity,” 'Vrlt," and "responsibility," and more recently the 
Secretary of Health, Education and Welfare, in emphasizing 

consumption ends of education, called attention to the importance of 

(13) 

"self-confidence," "curiosity," end "love of learning," ' Among the 
numerous measurements, some attributes receive the major attention, 
particularly "Perception of self" and some attitudes such as "perception 
of socictj'" in relation to self. Many worda clearly are used in describing 
the attitudes and attributes, but what is being identified on the one 
hand is the dignity of "being," the child as child into adulthood, and 
on the other is the attitudes toward society that make for motivation 



toward betterment. 
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Kumei'ous mcasuromcnto and i;ho ecalco on attrlbutco and attltudeo 

are available. Tlie ranp,o oC the studies carried out are oucseotcd by 

the chapter headings and chapter contents in tvjo handbooks of meaoure- 
(14,15) 

r.ients -- one on sociological nieacureinents and the other on 

measurer, lents of political attitudes, (E>:hibit8 I and II) 

The Social Sciences Research Center has compiled measurements to 

assess their usefulness and to v;ork tov.’ard definitional standardization 

as well as more widesp^read implementation of tested instruments, The 

American Psychological Association additionally has made technical 

(16) 

recommendations on psychological testing and diagnostic methods. 

« 

Considerable progress has been made toward uniformity and standardization 

of testing by development of the Minnesota multiphasic personality inventory. 

(17) 

In addition, Guilford and Zimmerman and alao Cattell have attempted 

to standardize concepts with respect to personality. To illustrate, a 
(18) 

handbook for the 16 personality factor questionnaire prepared by 

Cattell and Eber contains profiles describing personality test scores 

of largo samples in over 40 occupations. Evaluation techniques as 

they are applied in program analysis moreover have begun to be applied 

to mental health programming and to patient care in such a way as to 

encourage uniform design of criteria and uniform testing for evaluation 

purposes. (The program evaluation project of Hennepin County, Minnesota 

(19) 

is a major example.) 

In the work that Wayne Kimmel is undertaking at the Public Services 
Laboratory of Georgetovm University, he is reviewing the specific 

instruments that might be applied in measuring important constructs of 

/ 

attitudes and attributes. His research looks to recommendations on 
statistical data collection on attributes and attitudes. 
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2 , OiiCpiit; ScQucncc of Terforniniicfi nncl LontTC'f-rnn Product 
Education as a sequential and continuing process points to 
parallel nequonces , in outcome tneasuremont, •The multlproducto have 
somewhat different time dimensions, 

Tlie duality of purposes of education, or rather the possibility of 
considering the objectives of education (and measures of results) at 
different points in time is illustrated by 'Exhibit III, 

Follow-up Measurements — blngle-perlod performance accounting does 

« 

not adequately reflect child behavior. If many among those children tested 
happen to be on a temporary plateau in learning, the results of measured 
achievement may not fully reflect the preparatory absorption of knowledge 
that is on-going. Important questions are being raised about seauence and 
learning, Piaget's work has caused a reexamination of curriculum and 

(3 ) 

sequence. Bruner in- summarizing the work of a Task Force on 

Children's Learning emphasized the question: "I'/hat should be given 

r 

first, what next, and how shall a next step be influenced by how the 

child has proceeded up to here?" And he notes, "The questions, interestingly 

enough, remain recognizably the same whether one is asking how to arrange 

exercises for a kindergartener or freshman, how to program a teaching 

machine, how to organize chapters and sections of a book, how to present 
(4, p. iv) 

a ILccture," 

Presumably testing designs must follow on curriculum phasing step by 
step, phase by phase, just as curriculum must parcel out information to 
be learned. Scoring of that information oequired has to be defined and 
examined in small parcels. Programmod instruction Is an extreme of this 
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detnillnc find spccificntion of whnt la judged to be the leamlnf; requirement, 
with testing made n part of the learning experience Itcelf, For performance 
measurement, standardliccu tests may not alv/Ciya be regarded ao appropriate. 
Residual concerns in any case about national uniformity in testing will 
persist in sufficient force so that the result is multiple testing instruments, 
and perhaps for some time, continuing rollcnco on private agency rather then 
public testing or more statewide involvement in achievement testing. (20) 



^ In terms of both performance accounting end statistical measurements, 

the sequential facc'ts of education suggest folios;- through studies -- 

' .. ( 21 ) 

performance not for single years but periods of time. Rlvlln has urged 

that a longitudinal data system be developed to keep track of individual 

children and their fr4mtly background as they move through school, and for 

( 22 ) 

recording changes in their performance. She also has urged detailed 

program information at the level of the individual child to reveal, for 
example, not just whether he was in a school that had a remedial reading 
program but how many hours he spent in the program and the kind of program 
it was. Longitudinal studies are now being planned by the U. S. Office 
of Education to gather data on, among other things, educational outcome. 
Studies underway or planned call for data collection on high school 
graduation and also for first grade children. Much information is being 
sought about the child, including his prenatal care,' grades, and 
achievement scores.' The Intent of the early childhood schedule is to 
follow children through in grade levels 3, 6, and 8, as well as the 
first grade period. 

J./ Even in the context of upgrading of mathematics teaching, 
a longitudinal study was urged. 
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Propfirntoi'v vs. the pronont -- Tlie length of the nchooling period 



(one sixth to one qunrter of a lifetime) itself erguoo agninst exclusive 
concern v;ith p reparatory oducction. The investment or opportunity costa 
come to a substantial share of total living time. As o consequence of 
focusing at tha child and his moment of living in determining the product 
"education," much more weight comes to be placed on pleasurability , on 
feeling good, on being, and on joyful doing, (By and large, I am tending 
to use nonspecific psych 'cical or sociological tonns to make plain that 

I 

I have no expertise in thef. r use.) 

« 

Changing circumstances together with the advances in the processes 
of research in educational outcomes has led to a rediscovery of the 
consumption purposes of education. 

For one thing, the payoff for educational investment appears to be 
falling off. Differential years in school are- diminishing as the nominal 
average or median years of schooling is responding to automatic in-school 
promotions and the pressures against dropping out of school, and fewer 
and fewer persons remaining in the work force have only 8 years of 
schooling. A number of studies such as that of Sam Bowles of earnings 
differentials associated with skill differences show very large 

relative changes in skill content have been accompanied by very sir’ll 
earnings differences. For a subatantial period of years, despite the 
marked improvement in numbers of years of schooling, relative wages of 
college graduates or high school graduates (and the later vrages) or to 
that of 8th graders remained fairly constant. More recently, when 
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ncncrnl donumd has not been ovifficicnt to maintain hich employment, 
the stability of oamincs dif forontialc io v/cakoning, perhaps giving 
weight to the thesis of Ivor Berg's "Tlio Great Training Robbery" in 

which he attaches the productivity or rate of return from schooling 
arguing essentially that education io providing formol credentials of 
"progressively less economic importance," 



Mary Jean 3ov;man is one of the scholars' v?ho has moved the diocusnicn 
further by changing the concept of output from o skill learned to the 
outcome in interest in learning that skill. 



Only when we view men as men, not os bundles of skills, 
can we understand the role of education in economic growth or 
the effects of growth on incentives to undertake further 
education. Educotion becomes a means to enhance ability to 
learn end to adopt. 

Although there have been voices urging that the important 
contribution of education in production was specifically what 
it did to a man's ability to learn and to adapt in dynamic 
situations, little attention has been paid to this proposition 
in the cononical rate of return literature, (25) 



In the course of scientific inquirj^ of economic phenomena, a 
deeper penetration of educational outcomes has been made. 



If differences in skill levels acquired do not explain earnings 
variations, what does? Is It work motivation, or to use a Veblenian terra, 
"the instinct of workmanship"? Kelvin Lancaster, for example, has pushed 
the economist's view of consumption to examining the basic characteristics 
of the product relevant to the consumer, ' ^ 
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Traditionally education policioa have conaidcrod subatantivo 
education as preparatory for later livinp. Only with tho fornailation 
of the idea of human capital and investment in pcoplo, were financial 
and program policies of education redesigned so that they might bo 
more compatible with tho concept of education as preparatory. 

Economists began to apply the tools of their discipline and the 

concomitant measurements to education in the early 1960's, And in this 

application, the quantitative measurement of outcome for policy strategics 

essentially has its roots. Tlicodore Schultz initiated work on "Investment 
(27) 

in Man," And he contributed importantly to tho development of the concept 

of "investment" and the methods of measurcraent not only by his own 

research, but oy the research he encouraged. Gary Becker, in his volume# 

Human Capital . explored in greater depth than others the concepts 

of educational investment and the measurement of the rote of educational 
(28) 

returns. Becke : in a quantitative study restricted to white males, 

after personal taxes, shows a private rate of return from high school 
graduation rising from 16 percent in 1939 to 28 percent in 1958, Returns 
as measured are private rates of return and relate to earnings attributable 
to high school graduation, but do not take account of other future 
satisfactions. Others, including Mary Jean Bovraion, W, Lee Hansen, 

Burton Welsbrod, and Jacob Mincer are among those whose research deepened 
tho Concept of human capital formation through education and returns 
from the Inveatmcnts made, (27) 
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Edvicntioi'v ao an investment clearly has long-tctrm conscqucncco for the 
level and chanfjoc in economic groutli. Major studies of education os a 
source of growth and the measuremonto of outcome in growth hove been linked 
however, to the analyses of the "residual" or unescploined growth in GNP -- 
a growth not attributable to conventionally defined inputs. Edward F, 
Denison's important study of sources of growth quantified earnings 
differentials from years of schooling and the national income growth 
consequence of those added years of school, (Denison adjusts somev/hat 
arbitrarily for "ability" to get at the pure effects of schooling.) 

Denison's findings for the period 1950-62 are that education (again 

« 

measured by earnings differences attributable to years of schooling but 

adjusted downward for ability-effects) explain about 15 percent of the 

(29) 

total 3,36 percent per annum GNP growth rates. Jorgenson and 
(30) 

Griliches in a study of productivity change specified labor quality 

inputs into a nodel of growth in value added. In Griliches' study of 
manufacturing and determinants of state differences in value added in 
manufacturing using a labor quality variable, results showed education 
to have strong explanatory effects on value added. 

The longitudinal analysis of gross product from manufacturing 
found rates of growth in income 3,22 percentage points a year, with 
schooling per man accounting for ,73 percentage points or 23 percent. 

Labor quality is defin-^d essentially as educational inputs into the 
productivf system. And in the use of. education (years of schooling) as 
a proxy variable, new Issues come to be raised concerning the way in 
which education Impacts on growth or value added. (All these approaches 
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assume that years of schooling aio reflective of education no tho 

Investment and yields are assumed to bo enminga rofloctlve of 

(25, p. 118) 

productivity and productivity, of sUlll acquired,) 

Specification of raeaouremonts of invcatrrient outcomes and quantifica- 
tion of those outcomes in the past have aosigned little income return in 
investment in education of mothers who do not enter tho paid work force. 

The notion of investment in education for child motivation and learning 
generates still an additional postponed return. While explicit recognition 
of investment for parental involvement in tho education of the child 
has received little attention aside from somci work by Lovin^ Bowman, and 
this author, returns to the education of the woman who chooses to stay 
at home and rai'je her family is important for child learning and has a 
number of educational policy implications. 

In summary, those several approaches involve primarily determination 
of earnings differentials attributable to years (or quality) of education. 
And they point to a factoring out of "native" ability (without due 
regard to the creation of ability by education). While essentially the 
mcasurcmei'.ts developed in tho conceptual work that has been done is of 
the macro type that is not readily used in micro analysis requiring 
comparing of results from activity options for a school district or school, 
there are aspects of the measurements that are applicable to performance 
evaluation. Performance payments inen Investment context would mean 
payments for graduating a young person from high school, or payments 
related to some yardstick of skill created or earnings differentials 
created through educational services. 
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Co ndi i! I nr; Oli <irTv .''rJ cm 

The* built-in inccntlvo ntructvii'o of perfoj.'i.ir.ncG' contracting alioulcl liolp 
us i^uch l?no-.;lod'’e about the efficacy of contractual paymento an an 

incentive. Essentially vc are learning \;hethor performance contracting lo 
beneficial or, rather if it achieves \;hat io ileslgned to aciiic've, and to 
vhat degree or in vhat measure it clooc v/orlc; and under vhat circumstnncca . 
Performance rests on the outcome coug’’.''. Hud rcrmltc are the critical trstii'’ 
grounds. 

lihat criteria are to be used in measuring Duccecs? Tlie 4 A'o -- 
aptitude; achiovtmunt; attitude; and attribute -- presented in this paper 
are indicative of the multidiraencional products of concern in education. 

Most research in tlie past has centered on aptitudes and achievemento. Standard 
i;.ation of testS; Mpdating of their norras; and vidcr application to assure 
general usefulness continue to necessitate major additional efforts, Tlie 
achievement target of insuring a year’s progress for every year of school for 
every pupil is not possible of attaiiincnt. Some children cannot meet that 
target - v;ithout an inordinate input of resources. Further; programmed 
instruction has made plain norm referenced tests are not a substitute for 
the specifics of reading or numbering achievement; for example; learning a 
defined number of vjords of defined characteristics and designation. 

Much less developed are the measures of attributes and attitudes. 

Many different vacasuromonts have been formulated; and at least experimentally 
some have been tried on small population groups. National assessment has 
undertaken to include in its testing program attltudlnal tests related to 
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curriculum matter. But much rddttional rfoenreh io iipodcd, V?o can Btart to 
apply one or tv;o knomu tooting Inotrumontn on camplco of national population 
or on a state\;idc basic and improve the meaourcuicntc as experience io gathered. 
Or we can wait and opt for more research aa a prior condition to application of 
concept and inctruraer.t . A range of questions on both direct effects and on 
feedback;- could witli profit be tlic subject of research for purposes of fact 
gathering for educa':ional prograiiming, llovjovor;, a beginning on application 
may well clarify the need for and accelerate the rocearch on outcomes. 

Quality alone neasurod by the A's Is but one aspect among many of 
the complex product--educaticn. Quantities, while long familiar, are not 
unimportant. For e;rample, the numbers of high school graduates and numbers 
completing auto mcclianics training. Measurements of consumer voter, 
market-type decisions are indicative of consumer tradeoffs. And income — 
employment impacts significantly assess investment ends. Primary, secondary, 
and tertiary outcomes, both for consumption ana investment, coma into play. 
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LIFl- .SATISFACTIO:'! AND IIAPPrNi:SS 

Sbci.nl Svirvcys XnCo Life; Sntisffic (;lon 
Ro.l'i.c'ibilif.y of S.'jti.ijfacfion Moacurcu 
Corro'latca of Rcportoil SatiiuAicflon 
Rclafion Witih Other Attitudes 
Rcl.ntfon of Life Satirfoction to iJchavior 
Sumaiary and Goncliioions 



MFASURFI-IENT OF S£LF-*FSTE IM AflD RELATED CONSTRiJOTS 

1. Adjective Q-Sot for Non-Profoocional Sortt;:rs (Block 1961) 

2. Butlcr-)i.ciigh Q-Sort (Butler aiid Haigh 195A) 

3. Se.lf-conc.cpt Sc.nantic Differential (Sdivc'..-ta; and Tengri 1965) 
A. Self-concept Semrntic Differential (Per\'’iri and I illy 1967) 

5. Inventory of the Self Concept (Shervrood 1^62) 

6. Index of Adjustment and Values (Bills e^ 1951) 

7. Self-esteem .Scale (Rosenberg 1965) 

8. Personal Competence Scale (Campbell jojt nl^ 1960) 

9. Body-cathexis Scale (Secord and Jourard 1953) 

10. Low Self-esteem Scale (Hunt _et _al^ 1967) 

11. Self-acceptance f'calc (Berger 1952) 

12. Adjective Check List (Gough and Ileilbrun 1965) 

13. Self-esteem Inventory (Coopersmith 1967) 

14. Ego-atrongth Scale (Barron 1953) 

15. Interpersonal Chock List (Lear)' 1957) 

16. INs’enty Statements Test (Kuhn and McPartland 1954) 

17. Scale to Measure Internal-external Control (Rotter 1966) 

18. Multidimensional I-E Scale (Gurin _al 1969) 

19. Measure of Self-consistency (Gergen and Morse 1967) 



ALIENATION AND ANOMIA 



'1. linomy Scale (McClosky and Schaar 1965) 

2. Anomia Scale (Srolo 1956) 

3. Pov.’crlessness (Neal and Seeman 1962) 

4. Political Alienation (Ol.sen 1969) 

5. Alienation via Rejection (Streuning and Richardson 1965) 

6. Purpose-in-life Tost (Crumbaugh 1968) , 

7. Alienation Scale (Dean 1961) ' 

8. Alienation (Middleton 1963) 

9. Political Alienation (Horton and Tliompson 1962) 

10. Alienation (Nettlcr 1962) 

11. Anomie Scale (Hyman ejt £_1 I960) 

12. Helplessness (Gamson 1961) 

13. lilicnatiou (Davids 1955) 

14. Alienation Within a Social Syotem (Clark 1959) 
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; PUBLIC KE/iCTION TO GOVniL\MENT POLICIES 

* 

' Attitudes on V/ar 

Z' Relations Uith the Coranunist World 

National Defense and Arms Control 
Foreign Aid and World Trade 
Civil Rights 

Economic and Welfare Issues 
Liberalism and Consoi'vatisra 



LIBERALI SM-CONS 1 RVATISM 



1 . 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 
9. 

10 . 

11 . 

12 . 

13. 

14. 

15. 

16. 
17. 



Ideological Agreement With Goldwater (Selznick and Steinberg 1966) 
Conservntisn. Scale (McClosky 1958) 

Social Attitudes Scales (Ker linger ,1963) 

Conservatism-Radicalism Battcrj' (Centers 1949) 

Political-Economic Conservatism (PEC) Scale (Adorno et al 1950) 
Inventory of Social Attitudes (Eysenck 1947) 
Radicalism-Conservatism Scale (Nettler and Huffman 1957) 
Liberalism-Conser-vatism Scale (Wright and Hicks 1966) 
Liberalism-';onser\'’atism Scale (ICerr 1952) 

Liberalism-Conservatism Scale (Hartmunn 1938) 

C-R Opinionaire (Lentz 1935) 

Personage Acimiration (Lentz 1939) 

Situation-Response Survey (Pace 1939) 

Harper's Social Beliefs' and Attitudes Test (Boldt and Stroud 1934) 
Politico-Economic Radicalism-Consen^atism (Sanai and Pickard 1949) 
Radicalism-Conservatism and Social Kobillty (Hetzler 1954) 

Concern With Progress (Morgan, Sirngeldln, and Baen?nldt 1966) 



DEMOCRATIC PRINCIPLES 

1. Willingness to Tolerate Nonconformists (Stouffer 1955) 

2. Scale of Perceptioia of Internal Conununist Danger (Stouffer 1955) 

3. Democratj.c and Aiati -Demo era tic Attitudes ■ (licClosky 1964) 

4. Attitude Toward Democratic Principles (Prothro and Grigg 1960) 

5. Attitude Tounrd Communists (Schonbar 1949) 

6. Attitude Toward Civil Liberties (Noble erid Noble 1954) 



DOMESTIC GOVERNMENT POLICIES 

1. Domestic Social Welfare Scale (Campbell ^ I960) 

2. Attitude Toward Government (Opinion Research Corporation 1960) 

3. Big Business -Minded Scale and Socialism-Planning Scale (Rosenberg 1957) 

4. ' Attitude Tov;ard Socialized Medicine (ICubany 1953) 

5. Attitude Toward Government (Banerjee 1962) 
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ii/’vCiAL a:>’d irnnrcc attitudI'S 

1. Prejudice end Pvetionnllty (Scluunnn find Herdlnf; 196A) 

2. Xdcntlificfitn’.on V/ith the Underdof; (Sclninuin nnd Herding 1963) 

3. ^ rro~Intcgretinn Scnln (.'ihoateley 1966) 

4. ?Iul Lifactor Racial Attitude Inventory (Uoodinnnsee and Cook 1967) 

5. Social Die tancv: Sc/ilo (Rogardus 199/) 

6. EthnoccntrifiM Scale (Adorno a_l 1950) 

7. Racial Stereotype Indo: (Matthev/s and Prothro 1966) 

8. Racial Identification Index (Hatthevjf; and Protliro 1966) 

9. Co.T.aiunity Race Relations Ratings (Matthews and Prothro 1966) 

10. Dimensional Aetitude Measure Tov.'ard Negroes 0s^J^i8htcrann 1962) 

11. Prejudice Toia.rd Negroes (Hostie 1953) 

12. Paired Direct and Projective Questionnaires (Gcttela and Ualsh 1958) 

13. Attitude Toward the Negro (Hinckley 1932) 



HOSTILITY-RELATED l^V.TIONAL ATTITUDES 

International Reactions Scale (Christiansen 1959) 

National Patriotism Scale (Christiansen 1959) 

International Hostility Inventory (Grace 1949) 

Attitudes Tovnrd War (Putney 1962) 

Vietnam Policy Scales (Verba £t a_^ 1967) 

Ideological Militancy-Pacifism Scale (Dombrose and Levinson 1950) 
Hostility in Interne tional Relations (Helfant 1952) 

Nationalism (Perguson 1942) 

Nationalistic Attitude Changes (Stagnor nnd Osgood 1946) 

Attitude Toward War (Day and Quackenbush 1942) 

Attitude Toward War (Stagner 1942) , 



COMMIIvTTY -BASED POLITICAL ATTITUDES 

1. Community Attitude Scale (Bosworth 1954)' 

2. Local-Cosmopolitan Scale (Dye 1966) 

3. Cosmopolitanism Scale (Jennings 1965) 

4. Localism-Cosniopolitranicia Scale (Dobrlner 1958) 

5. Acouaintanceship Scale (Schultze 1961) 

6. Attitude Tov.’ard Sources of Pov;er (Hnar 1956) 
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2 . 

3.. 

4. 

5. 

6 . 

7. 

8 . 
9. 

10 . 

11 . 




POLITICAL PARTICIPATION 

/ 

1. Political Participation Scale (Matthews and Prothro 1966) 

2. Political Activity Index (Woodward and Roper 1950) 

3, Index of Political Participation (Campbell ^ a_l 1954) 

4, Political Participation (Robinson 1952) 
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5. rublic Affalrr; Opinion Lenclci'v';liip (iral:;: ;-.nd Luirarafold 19r)5) 

6. Opinion Leadership (Lasiirsfeld, )\cro]!ion, and Gnndet 19'^14) 

7. Opinion Leadership Scale (Rogers 196/!) 



Ami'ULCS TOWARD T!;E POLITICAL PROCESS 



1. Subjective. Political Coir.petenco Scale (Almond end Verba 1963) 

2. Various Otlier Attitudes About the Political System (Almond and Verba 1963) 

3. Index of Ratio of Support (McClosky pj: a_l 1960) 

4. Political Involvement (Campbell e_t ni_ 1960) 

5. Political Efficacy (Ca;;:pb'!ll a_i 1954) 

6. Sense of Citizen Duty (Cvimpbell c_t n_l 1954) 

7. Extent of Issue Orientation (Campbell e^ 1954) 

8. Issue Involvcii.ent (Campbell jet a_l 1954) 

9. . Issue Partisarship (Campbell ej: ^ 1954) 

10. Overall Indejc of Psychological Rcadineso for Participation 
(Matthev;s and ProthroJL9l3^) 

11. Sense of Civic CprrtpftTtence Index O'R'^tthews and Prothro 1966) 

12. Parlg,?' Imj£i>^coro (Matthews and Prothro 1966) 

13. Dimensions of Political Norms (Litt 1963) 

47^ Political Cynicism and Personal Cyniciora (Agger, Goldstein, and 

Pearl 1961) 



AUTHORITARIAiaSM, DOGMATISM MD RELATED MEASURES' 

1. The California F Scale, (Adorno £t a_l 1950) 

2. A Kew F Scale (Iv^ebster £t _al 1955) 

3. Forced Choice F Scale (Berkowitz and Wollcon 1964) 

4. Forced Choice F Scale (Schuman and Harding c. 1962) 

,5. Balanced F Scale (Athr.nasiou 1968) 

6. Shortened F for Political Surveys (Janowitz and Marviclc 1953) 

7. Four-Item F Scale (Lane 1955) 

8. Ten-Item F Scale (Survey Research Center 1952) 

9. Pensacola Z Scale (Jones 1957) 

10. Fiiscist Attitudes Scale (Stagnor 1936) 

11. Unlabeled Fascist Attitudes (Edwards 1941) 

12. Anti-Semitism (Levinson and Sanford 1944) 

13. Traditional Family Ideology (Levinson and Huffman 1955) 

14. Status-Concern Scale (Kaufman 1957) 

15. * Rigidity Scale (Rehfisch 1958) 

16. KAPII Scale (Meresko et al 1954) 

17. Rigidity Scale (Wesley 1953) 

18. Intolerance of Ambiguity (Rudner 1962) 

19. Intolerance of Ambiguity (Martin and Uestie 1959) 

20. Desire for Certainty Test (Brim 1955) 

21. Ethnocentric Democracy Scale Otyi^an ejt nl_ 1962) 

22. Toughmindednoss (T) Scale (Eysenck 1954) 

23. Dogmatism Scale (Rokeach 1956) 

24. Short Dogmatism Scale (Schulze 1962) 

25. Short Dogmatism Scale (I’roldahl and Powell 1965) 

26. Opinionation Scale (Rokonch 1956) 

27. Intellectual Conviction Scale (Rokeach and Eglash 1956) 



OniER SOCIO-POLmCAL ATL'XTimKS 



1. National Involvomont Scalon (DcLnmntor qt 1968) 

2 . Beliefs About the Dlstrihution of Power (rnrin and llytlna 1969) 

3. Soda] Ivcsponclbi'ilty Scale (Beirkov^itj; and Luttorr.ian 1968) 

4. New l.eft Scale (Ciiriatlo et _nl 1969) 

5. . Nadicallsm-Coiu;ci*\'atisra (Co;;irGy and Ncvancycr 1965) 

6. Inputs to the Political Syatc.",'.! 0‘illjrnth 1968) 

7 . Opinion Leadorshlp (Scbciich 1960) 



VALUES 

1, Study of Values (Allport, Vernon, and Llndzey 1960) 

2, Test of Value Activities (Shorr 1953) 

3, Survey of Ineerpcrsonnl Values (Gordon lS-60) 

4, Personal Value Scales (Scott 1965) 

5, Value Profile (Bales and Couch 1969) 

6;A Dimensions of Values (Withey 1965) 

7, Cliange<s in Moral Values (Rcttig and 'Pas am; mi clc 1959) 

8, Inventory of Values (Ewell 1954) 

9, Value Survey (Kokeach 1968) 

10. Mays to Live (Morris 1956) 

11. Variations in Value Orientations (Kluckhohn and Strodtbeck 1961) 

12. Social Values Questionnaire (Porloe 1967) 



GENEIU\L ATTITUDES TOWARD PEOPLE 

1, Machiavellianism (Christie and others 1969) 

2, Philosophy of numan Nature (Wrightsman 1964) 

3, , Faith in People (Rosenberg 1957) 

4, Trust in People (Survey Research .Center 1969) 

5, People in General (Santa 1961) 

6, Misanthropy (Su]livan and Adolson 1954) 

7, Acceptance of Others (Fey 1955) 
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Ill, Sutninary Classification of Outputs 
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Knip 1 oNancnL Relations Under Performance Contracting* 

Dr, Myron I^Icberman 
City University of New York 

In a paper dealing with policy is;sues, it is often helpful if an 
author reveals the biases, however tentative they be. In this case, I do so 
gladly; my view is that objectivity consists not in concealing one's real 
preferences, bat in articulating them fully, so that others are better able 
to evaluate their impact on the conduct of a study. With this in mind, let me 
state my initial biases relating to emplo)on€'.nt relations and performance 
contracting. 

1 . Performance contracting is a potentially significant innova- 
tion in education, with better long than short term prospects for bringing 
about basic reforms in education. 

2 . Collective bargaining and stiong teacher organizations are 
desirable as a matter of public policy. The rationale for them is stronger 
in public education, and in public employment generally, than it is for 
collective bargaining and strong unions in t:he private sector. This ration- 
ale is essentially a conservative view, having little or nothing to do with 
the impact of collective bargaining upon the salaries or benefits of the 
employees concerned. In my view, arbitrary or discriminatory governmental 
action is an ever-present danger . Individual citizens or individual public 
employees are typically ineffective in resisting such undesirable govern- 
mental action. Unless there exist strong organizations, relatively free 
from governittc.pt control, and rer3dy, willing, and able to require public 
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ofliclals to explain and defend their actions, there are unacceptable risks 
of inefficient, cori’upt, and discriminatory government. Such organisations 
y on occasion prove to be a troublesome obstacle to desirable government 
action, but the advantages of having S’:h organizations far outweigh the 
disadvantages , 

3. In the long run, the extent to which performance contractin>> 
is used, and fulfills its potential, will de))end upon the effectiveness of 
public management. The opposition to perfonaance contracting by teacher 
organizations may be a decisive factor in its rejection by some districts, 
but over the long haul, the future of perfomance contracting is up to 
school management, 

4, Employment relations is an extremely important dimension of 
performance contracting. In fact, this assuiiption was a decisive factor ia 
my decision to investigate the topic. 

My sympathetic view toward performance contracting has many roots. 
As a resident cf New York City, I pay more in state and local taxes than I 
would anywhere else in the United States, Wien the city administration 
asserts, as mine has lately, that sanitation workers in the private sector 
are three times as efficient as those in the uniformed services, and that 
the city could save a ereat deal by contracting out garbage collection, my 
reaction was "Why not do it, instead of just talking about it?" 

A fev years ago, a close friend of mine accepted employment with 
one of the country's leading land developers. In discussing this firm, my 
friend was especially impressed by the effectiveness of the firm's proce- 
dures for deciding whether or not to contract out certain tasks, such as 
pub] ication i , Since then, I have been especially interested in school board 
policies on the subject. My impressionistic judgment is that very few 



162 



3 



.school hoiirds, if ;my, have thouphi Irut subcontract Inp, systematically. 

Work sucli ns running a cafeteria, busing students, or maintenance is some- 
times contracted out, but system operations are not analyzed systematical Iv 
lor their contracting out possibilities. Since policies which have not been 
analyzed critically often provide fertile ground for reform, I was - and 
still am - prepared to accept the possibility of dramatic improvements 
through some form of contracted instructional services* 

Fianally and to remove any possible doubts on an attitudinal 
matter which often dominates current discussion of performance contract- 
ing let me repeat a comment which I have repeatedly affirmed since it 
first appeared in print over ten years ago: 

**In one sense, I place myself with the 'critics* of public educa- 
tion. I am convinced that our schools are not as effective as they 
should be. I mean by this more than the simple idea that improve- 
ment is possible. Improvements are possible in every social insti- 
tution — our courts, our legislatures, our hospitals, and so on. 

When I cast my lot with the 'critics,' I mean to say that the return 
on our educational investment is too low to be brushed aside by sin- 
cere but routine admissions that improvement is possible and desirable. 
I am asserting that the gap between the achievable and the actual re- 
sults of public education should be a matter for deep national concern 
and that this concern must not abate until the gap has been drastic- 
ally reduced.** 

Myron Lieberman, The Future of Public Education 
(Chicago: The University of Chicago Press, 1960), p, 13. 
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^ ^ • l^roc:o. i ural prob 1 cms 

Like some performance contractors, It appears that my performance 
was not equal to my intentions. This paper admittedly lacks the empirical 
base for an adequate assessment of employment relations under performance 
contracting. 1 have read performance contracts, position papers, and arti- 
cles on the subject. Over the past tvo weeks, I have talked to a variety 
of sources on a variety of issues relating to employment relations under 
performance contracting. Unfortunately, the task proved to be much more 
complex than was originally anticipated. 

For example, in one district, a teacher or^^anlzatlon representa- 
tive presented me with documents and arguments to the following effect: 

1. The superintendent v/anted to participate in the OEO projects 
to enhance his image as an innovator, and thereby persuade a new board to 
retain him. 

2. His successor, who was black, embraced the project as a source 
of patronage, loading the staff with unqualified persons. 

3. The project was advocated publicly as a means of getting rid 
of, or overcoming, white teachers who didn’t care about black students. 

4. Teacher representatives coming to a meeting to resolve the 
problems were iiiet with pickets labeling the teacher representatives as 
racists. 

Now these and other allegations are serious matters. However, in 
order to assess them adequately, and to be fair to others involved, it would 
have been necessary to interview the district board and administrators, the 
project staff, the paraprof esslonals employed, teachers in and out of the 
project, OEO personnel, and others in a position to contribute to an object- 
ive assessment. There were literally dozens of situations where such pro- 
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ccdurcs would have been required to deal with the gut issues. Unfortunately, 
1 was not able to uLilize such procedures, much as I would have preferred, 
and as valuable as the study might be if conducted this way. 

The difficulty of getting the facts was not confined to assessing 
conflicting accounts of empirical events. For what it may be worth, only a 
few school districts, contractors, and/or teacher organizations returned 
copies of the ])erformance contracts. I recognize the time and effort which 
may be involved, especially since the parties undoubtedly received many such 
requests, but the impression left is hardly an attractive one. When school 
districts send fancy brochures extolling the virtues of a performance con- 
tract, but cannot seem to find copies of the contract itself, which is a 
public document, some skepticism is inevitable. This is especially true 
when the demand for copies should have been anticipated. 

Another problem is that there is so much discussion of perfor- 
manv^e contracts in terms of assumptions or specific practices which are not 
inherently related to performance contracts. For instance, consider the 
following quotation from a position paper on performance contracts: 

**One of the manifestations of the current call for accountability 
in the public schools is thr> concept of performance contracting. It 
is most simply defined as an educational experience in which a private 
corporatiou contracts for classroom instruction.** 

A Statement of Position and Suggested Guidelines on Performance 
Contracting , adopted by the 1971 Spring Representative Assembly 
(East Lansing, Michigan: Michigan Education Association, 1971), p. 1. 
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This qu(;L;ilion confuses the concent of nerformnncc contracts with 
the concept of subcontracting, a very pervasive and unfortunate confusion in 
the. literature aiul discMission of performance contracting. Some of the most 
perceptive and most adamant opponents of performance contracting per so 
explicitly stated that their opposition was not to subcontracting but to 
performance contracting, 

III. IssuCvS in employment relations under p erformance contracts 

Let me now turn to some of the basde issues relating to employ- 
ment relations under performance contracts, or as they might be affected by 
performance contracts. Some of these issues antedate nerformance contract-- 
ing, but performance contracting has generated greater interest and greater 
pressure to clarify them, 

1* Merit pay 

Merit pay is defined here as a procedure for compensation 
v/hich distinguishes between two or more individuals doing the same kind of 
work on the basis of real or alleged differences in the quality or the out- 
put of the work performed. Salary differentials based upon different job 
descriptions, such as would be forthcoming under differentiated staffing, 
are not "merit pay" as just defined. 

Teacher organizations have been opposed to merit pay for 
many decades. It is not surprising, therefore, that they opposed perfor- 
mance contracting on this basis. Nevertheless, logically and practically, 
the issue of merit pay for individual teachers or other employees under 
performance contracts is irrelevant to the concept of performance contracts. 
That is, performance contracts can be introduced in a 

meaningful way without merit pay. Of course, a district may wish to introduc 
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pcrCormcincc contracts in a way wliicli includes merit pay. Unfortunately, 
however, evils or alleged evils of merit pay have already been tied to 

performance conlract?; per se in ways that could and should have been avoided. 

In this connection, some of the supporters of performance con- 
tracts have been their own worst enemy, insofar as employment relations are 
concerned. Compensation for policemen, firemen, sanitation workers, and a 
host of other public employees (by far the majority) are not geared, collec- 
tively or individually, to results or to merit pay, as is envisaged by some 
kinds of pcrforniance contracts. Furthermore, it is fallacious to assume 
that most employees in private employment or the professions are paid this 
way. For example, the vast majority of employees in unions (about 20 
million) are paid on the basis of services as rendered, not on the output 
or benefit of services as received. 

The professions present a mixed picture. If your law>^er wins a 
case with a large award, he is apt to bill you on the basis of the benefits 
actually received by the client, i.c., services as received. If you lose, 
the lavTyer doesn't; he bills you for his time, i.e., for his services as 
rendered. Similarly, a physician may charge a great deal for a little 
effort when he saves a life ^y fast action. By contrast, when the patient 
dies, the physician merely shifts his bill to a services as rendered basis. 

In any case, it is inaccurate to assert that most people in our 
economy are paid according to results. Perhaps they should be. Be that 
as it may, I am already unequivocally on record to the effect that compensa- 
tion in education can and should be related to productivity in a meaningful 
way. Hov/ever , I question whether the way to do this is to give the impres- 
sion that teachers are the major exception to the principle of basing com- 
pensation upon productivity, or that teacheriJ constitute an exceptional 






pocket of rcsi£;tance to the idea. Furthermore, there is a world of 

difference between gearing the compensation of a 
group of employees to the productivity of the grouo, and gearing the compen- 
sation of individual employees to individual measures of productivity. Some 
teacher opposition to performance contracting (how much is difficult to say) 
could (and in iry judgment should) have been avoided by a clear-cut recogni-- 
tion of the distinction at the outset, coupled with an explicit avoidance 
of merit pay. Such avoidance need not have been permanent for 

those who regard merit pay as vitally important. As matters stand, however, 
there is nothing in the theory or practice of merit pay under performance 
contracting which differs significantly from approaches already rejected by 
teacher organisations, as by most organized employee groups. 

The preceding comments should not be interpreted to mean that 
merit pay is a permanently hopeless proposition. T would like to suggest, 
however, that l.he basic problems of teacher comoensation lia elsewhere, an 3 
that performance contracts .. at least in their present form, are not likely 
to contribute nuch to the resolution of these problems. Let me elaborate 
on this point briefly. 

A physician who does not keep abreast of developments in his 
field runs serious risks. For example, if a certain drug is found to have 
harmful effects in certain kinds of patients, and the fact is publicized in 
the medical literature, the physician may be guilty of negligence or mal~ 
practice for failure to act on the information. Similarly, la^o^ers and 
accountants are under constant pressure to keep abreast of developments in 
their fields. Even where malpractice is not an issue, the relationship 
between the prof essional ’s information and his appeal to clients is too 
obvious to be ignored. 
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In oduceTtion, there is no such pressure on teachers to stay 
abreast of their field. The overwhelming majority of teachers are on a 
salary schedule and paid on the basis of years of education and experience. 

A teacher can continue to use outmoded methods for years without the slight- 
est pressure from anyone. The problem is not that teachers don't care. It 
is that compensation structure of education dees not offer the rewards, and 
does not put as much pressure, or pressure of any kind, on teachers to stay 
abreast of professional developments . 

Charging this structure will be an enormously difficult and com- 
plex task, to say the least. Prima facie, it seems that the change would 
have to be preceded by a clear-cut demonstration of the superiority of teach- 
ers and of instructional systems which utilir-e v/hat is known about the way 
to get the job done. Demonstrations which bog down in the finer points of 
measurement are not likely to provide the political base which is necessary 
for effective Innovation. 

None of the performance contracts known to me really addresses 
itself to this structural problem. In fact, they may even 
exacerbate the problem in the long run. At best, successful contractors 
will turnkey in the optimum level of expertise at any given time. This is not 
to be denigrated, especially in a field where so many practitioners perform 
at a primitive level. The problem is that in the absence of a continuous 
relationship between the professional's information system and his compensa- 
tion, what is turnkeyed is likely to prevail in the system for a long time 
to come. This is fine as long as new and better procedures are not devel- 
oped. Suppose, however, that after all the travail in a particular district, 
there is improvement and something g^ts turnkeyed into the system. Suppose 
further that something even better comes down the pike the following year. 
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Will the superintendent go to the hoard to reauest a new performance contract 
contemplating a new turnkey operation? I doubt it very much. 

Perhaps this is one of those problems, like how to spend a million 
that we would like to have. Regardless, let me summarize by saying that it 
was unwise to inject the issue of merit pay into performance contracting. 

On the record to date, insofar as it is available to me, performance con- 
tracting has only kicked a sleeping dog, better left alone at this stage of 
development, in its approach to merit pay. 
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2 . Differentiated staffing 



One of the basic weaknesses in the educational personnel 
structure is its lack of differentiation. Teachers differ by subject 
matter specialization; otherwise, there is relatively little differentia- 
tion by function (diagnostician, remediation, clinician, measurement, etc.). 
The teacher aide project at Bay City, Michigan in the 1950’s was the 
forerunner of recent efforts to introduce some meaningful improvements in 
the educational personnel structure. "Team teaching" is a more recent 
effort characterized by considerable publicity and infinitesimal substance.. 

Performance contracts could move us much closer to developing e 
more productive personnel structure. However, if it is to avoid becoming 
another fiasco in the teacher aide and team teaching tradition, the use of 
paraprofesslonals under performance contracts will have to follow a differ- 
ent path than the one taken to date. The different positions in the employ- 
ment structure will have to be defined much more carefully, and related tc 
training programs, educational facilities, and schedules in a rational way. 

Thus far, no such development seems to be emerging from perfor- 
mance contracting. The projects under way present a mix of prof esslonal- 
paraprof esslonal relationships, but none seems to be making it as a perma- 
nent feature of the educational personnel structure. At the same time, 
however, it is already apparent that any such development will present seme 
extremely difficult problems of emplo 3 nnent relationships. Let me list just 
a few of the basic questions which have already arisen and which would become 
matters of basic public and organizational policy if we develop a differen- 
tiated personnel structure. 
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1) Should paraprof essionals have to be licensed? 

2) To what extent should paraprof essionals receive the same 
fringe benefits as teachers? 

3) Should paraprof essionals be in the same bargaining unit 
as teachers? 

4) Should paraprofesslonals be represented by the same bar- 
gaining agent as teachers? 

5) Should organizational eligibility be the same for para- 
professlonals and teachers? 

The way in which such questions are resolved is of the utmost import- 
ance to education. Their importance is somewhat obscured where the contractor 
employs the paraprof essionals , but if a process is turnkeyed, the issues would 
emerge very quickly. 

Significantly, the AFT and NEA are following different paths on 
this issue. The AFT is already making an intensive, and fairly successful 
effort to organize paraprofesslonals. Paradoxically, this approach was a 
consequence of the far reaching confrontation between the UFT and certain 
black groups in Ocean Hill-Brownsvllle in 1968-69. On the other hand, the 
NEA has not made a firm decision at the national level to organize nara- 
prof essionals; for a variety of reas6ns, there is more reluctance in NEA than 
in AFT to having the teacher organization organize paraprofesslonals. 

Consider some of the ramifications of this situation. 

1) If paraprofesslonals are not Included in a bargaining 
unit, management is free to define the job and the terms and conditions of 
employment at management's discretion. If the paraprofesslonals are in a 
bargaining unit, management has to bargain over terms and conditions of 
employment for paraprofesslonals. 
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2) If paraprof essionals are in the same bargaining unit as 
teachers, what will be the implications and ramifications for membershln in 
teacher organizations, in the racial composition and policies of the teacher 
organizations, and in theobjectives of the combined organization at the 
bargaining table? 

If you are interested in some of the dimensions of these problems, 
consider the following facts. In 1961, the UFT had about 2,500 members. 

In 1971, it was the largest union local of any kind in the United States; T 

have been told that the UFT is the largest union local of any kind in the 

( 

world, and this may well be the case. It is already larger than many inter- 
national unions affiliated with the AFL-CIO, and it is probably going to get 
much larger in the near future. 

For the most part, paraprofessionals employed by contractors have 
not posed serious problems for the bargaining agents. Some contractors have 
made a point of paying teachers and paraprofessionals the same as called for 
by the regular district contract. The point is, however, that the scope of 
unbridled managerial discretion relating to paraprofessionals is rapidly 
shrinking. Furthermore, we can expect the costs for paraprofessionals to 
rise in the near future, since their organizations will be under strong 
pressure from several sources to achieve dramatic gains for them. 
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motive 

Lenders of teacher organizations typically get and keep their 
positions chieHy because of their ability to achieve material benefits for 
their constituents - or to persuade them that this is hapoening, whether it 
is or not. We need not worry about the charges by such leaders that wanting 
to make money is an illegitimate motive to be in the the education business. 

It may be a mistaken one, but it is no more inappropriate for a large corpora- 
tion to enter the education market for profit, than for a person to decide to 
become a teacher - or teacher organization leader - for this reason. In 
short, criticism of performance contracts because the contractors want to 
makeprofits is irrelevant. 

In this connection, a study of Peace Corps members has some 
relevance. The study showed that members of the Peace Corps who joined be- 
cause they saw something in it for themselves, as well as for humanity, typi- 
cally served mere effectively than those who joined solely for altruistic 
reasons. This makes sense and perhaps would not justify notice, except in 
education where there is so much overemphasis on motives for becoming a 
teacher and so little attention to the factors that actually shape behavior 
on the job. 

Of course, wanting to make money can lead to undesirable behavior. 

A restau.rant owner may serve stale food. An automobile mechanic may get you 
to buy new parts that are not really needed. Lawyers may urge you to sue 
when you should settle, or vice versa. In all these and other situations, 
the sensible course is not to deny the dangers but to take the practical 
actions required to minimize them. Of course, the cost of protection against 
the dangers is a legitimate consideration in assessing a system of control, 
but I sec no reason to regard the problem in. performance contracting as too 
difficult or too costly to solve. 




Of course, a common argument Is that the profit motive leads 
contractors to make all sorts of promises that cannot be fulfilled. They 
raise false hopes and then resort to shady tactics or downright dishonesty 
(e.g., teaching to the test) to cash In on their promises. 

Careless use of the term "guarantee" Illustrates the way In 
which supporters of performance contracts have strengthened the credibi- 
lity of Its oppanents. If there Is a "guarantee," It Is not that students 
will learn but that the contractor will not be paid If the students do 
not learn. One can argue that the typical "money back guarantee" In 
the commercial world means only no payment If there Is no performance, 
but this Is Ingenuous, to say the least. At least. It is If we are 
talking about an experiment Instead of a tested product. The semantics 
may have been effective In generating school board support, but they 
certainly have had a negative effect upon teachers. 



16 




Unfortunately, the invalid criticisms of performance contracting 
because it is based upon the profit motive have obscured some of the basic 
dynamics of the process. We, or at least I, do not know enough about how 
the contractors hope to profit from performance contracts as a long range 
enterprise. Some contractors appear to regard performance contracts as a 
means of selling Instructional materials in which they have a proprietary 
interest. Such a view might induce firms to assume larger risks than if 
each performance contract had to show a profit in its o\\m right. From the 
standpoint of enployment relations, these alternatives might affect the 
salaries or incentive payments to school district employees. In any case, 
we need to know a great deal more about the total economics of the situation 
(as distinct from the project account per se) from the contractors’ point of 
view. Otherwise, we are not likely to achic'^'e a realistic understanding of 
employment dynamics under performance contracting. 

A related issue is what happens to the rates if contractors or 
Individual employees make large profits or incentive payments. Such an out- 
come would undoubtedly create pressures to raise the required performance 
levels, both at the contractor and the Individual employee level. Under- 
standably, the setting of rates for bonus or Incentive or even regular pay 
is a sensitive issue in employment relations; the parties are continuously 
striving to have them adjusted or readjusted to advance their interests. In 
education, little attention has been paid to this issue, perhaps because of 
the turnkey assumption. If school districts are going to incorporate the 
practices demonstrated by the performance contracts, and do so witVi their own 
employees, there is no need to be concerned about the size of the incentive 
payments. Furthermore, many districts are participating only because their 
own money is not Involved. Under such circumstances, the amount of profits 
is a state or an OEO, not a local concern. 
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The individual incentive payments present some different problems, 
but I encountered a surprising lack of concern about the rates involved. I 
attribute this not so much to the turnkey aspect as to the fact that perfor- 
mance contracts involve funds which the districts would otherwise not have. 
Otherwise, there would be more concern about the rates. 

Regardless of the success of current projects, measured in terms 
of pupil gains, there are difficult questions concerning what is to be turn- 
keyed. Perhaps this is simply a deficiency in my understanding or my re- 
search, but it is not at all clear to me what would be turnkeyed, assuming 
that districts wished to incorporate whatevei led to “success” in the pro- 
jects. It would be helpful if this were clarified in some way that went 
beyond vague generalities. As it is, the observation that it is not “the” 
system but the fact that the contractors hav(; “a” system may have consider- 
able validity. 
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^ • l« cg i slntive reform 

It is nl ready apparent that the laws requiring the presence 
of certificfi teacliers have been an obstacle in some performance contracts. 

In my judgment, it is desirable tliat we revise the state laws requiring 
that certified teachers supervise students fo?; the school day ~ or alter-- 
natively, enable! persons certified at much lower educational levels to 
supervise children. Where the laws cannot be repealed, they should at least 
be amended to provide long range experimental situations involving non- 



certified personnel, * Teacher organizations will undoubtedly oppose such 
changes. For example, the policy statement o:3 the Michigan Education 
Association on performance contracting states that: 



**Teachers, along with all other instructional personnel, must be 
appropriately certified in accordance with Michigan law," and "Students 
participating in any performance contracting project must be subject to 
ail laws, rules, and regulations which pertain to attendance and pupil 
management in the state of Michigan." 

A Statement of Position and Suggested Guidelines on Performance Con- 
tracting (East Lansing, Michigan: Michigan Education Association, 

1971), p. 3. 



Nevertheless, teacher opposition to laws requiring that pupils be 
under the control of certified personnel during the entire school day is on 
weak theoretical as well as practical grounds. According to the conventional 
wisdom in these matters, the "professional" part of the teacher's task is 
"instruction," This view is usually accompanied by a very elastic and broad 
definition oi "instruction," such as in the following statement by the AFT's 
Director of Research; 
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r would maintain that those tasks which relate to or involve learners 
in any way are, in essence, instructional. If a person pertorms such 
tasks as grading multiple-choice or true/false tests, maintaining order, 
and supervising children, he is performing instructional tasks, and, in 
effect, is a teacher of children. .. In defining and, hopefully, stabiliz- 
ing the concept of a teacher, 1 would submit that a person involved with 
children to the extent that he has the ooDortunity to influence behavior 
operates, in essence, as a teacher.” 

Robert D. Bhaerman, Education * s New Uuali sjn^s (Washington: /jnerican 

Federation of Teachers, n.d.), p.3. 

The preceding quotation illustrates a point of view which should 
be corrected promptly. To see why, ask yourself what is the ’’professional” 
work of a physician Sometimes it is diagnosis. That is, it may require the 
utmost skill and knowledge to diagnose an ailment. Onc:e diagnosed, however, 
it may be that a nurse or even a secretary or the patient himself can carr> 
out the appropriate prescription; e.g., taking one tables every four hours. 

In other situations, diagnosis is obvious, even :o 1 \.k layman. 

The patient’s arm is broken or his skli is burned. In such cases, th^ "pro- 
fessional’* role, i.e., the one requiring expert skill and judgement, is not 
diagnosis, which may be obvious to everyone, but prescription or implementa- 
tion of prescription. A technician is often able to test for cancer, but a 
physician is required if an operation is required. 

It wouj.d be folly to decide apriori and apart from the realities 
of certain tasks what requires a trained physician and what can be do?ie by 
nurse’s aides, or by other paraprof essionals. Clearly, it would be foolish 
to decide that diagnosis must always be conducted by, or in the presence of. 
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a physician. Similarly, it would be unwise to say that prescription must be 
condiictcd in such presence. Yet this is what we have done in education, with 
far less justification than our much looser controls over medical activities. 
Thus there situations wherein expertise is required to diagnose an educa- 
tional deficiency; once diagnosed, however, a paranrof essicnal or ay intelli- 
gent lay person can sometimes provide the remedy, e.g., simple drill on a 
particular skill. There are other situations in which the educational diagno- 
sis may be obvious, but the remedy requires a very high degree of expertise. 
For this reason, we should avoid legal definitions of "teacher” which assume 
that their expertise necessarily falls in the area of instruction. By the 
same token, we ought also to drop these all-e)>bracing definitions of "instruc- 
tion." which seem more des-^gned to protect teacher employment than to any- 
thing else. 



Even if the case for performance contracts is weak 
or nonexistent, a strong case remains for modifying the legislation requiring 
that teachers supervise students at all times. I believe, however, that such 
changes could be advocated in a way that would generate the support of teachers 
and teacher organizations, (or at least neutralize their oppostion), despite the 
maladroit way the issue has been publicized thus far. 

In this connection, a more realistic attitude toward the custodial 
fundtions of schools may be in order. No one prefers schools to be merely custodial 
institutions, even for a limited part of the school day. Nevertheless, we 
may do better educationally by accepting some custodial functions for what 
they are than by continuing to insist that every second of every school day 
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will be nn enrichinR educational experience if only under the guidance of a certi- 
fied teacher. It is conceivable that children would be better off educa- 
tionally at age 6 if ages 3 to 5 had been spent In a center which was less 
than an elementary school but more than a babysitting operation. Neverthe- 
less, we cannot just superimpose such centers uDon the existing structure. 
Conceivably, we migiit finance such centers, at least in part, out of savings 
made possible by reducing the amount of unnecessary time elementary puoils 
are under the supervision of certified teach(irs. Experience in Canada and 
some other countries, with educational and li'.teracy standards equal or super- 
ior to our o\>m , but with significantly lower certification requirements, 
especially at the elementary levels, provider, additional support for this 
argument. And to avoid an error which is frtiquently attributed to others in 
this paper, I hasten to add that excessive credentialllng or licensing require- 
ments characterize a wide range of occupations and are a pervasive phenomena 
in our society. Licensing requirements geneially are coming under widespread 
legal attack for their discriminatory effects; in fact, OEO ought to explore 
this matter vigorously as a part of its ongoing ml ,ion. 
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At Llio present time, n majority of the nation's teachers work 
pursuant lo a collective bargaining agreement. Nevertheless, very few of 
these agreements specifically refer Lo performance contracting or the con- 
tracting out of work. If experience in the private sector is any guide, 
this is likely to change. References to contracting out are very common 
in collective bargaining agreements in the private sector. In addition, 
there is a wealth of literature on the subject; a recent bibliography on it 
runs to 58 pages. 

David M. Farrell, The Contracting Out of Work : An Annotated Biblio- 
graphy (Kingston, Ontario: Industrial Halations Centre, Oueen*s 

University , 1965) . 

The few agreements in public education which do refer to the 
contracting out of work support rather than restrict management's rights 
to contract out. The following clauses are illustrative: 

** The Board’s right to operate and manage the school system is recog- 
nized, including the determination and direction of the teaching 
force, the right to plan, direct, and control, intlate or discon- 
tinue school activities; to schedule classes and assign workloads; 
to determine teaching methods and subjects to be taught; to main- 
tain the effectiveness of the school system; to determine teacher 
complement ; to create , revise, and eliminate positions ; to estab- 
lish and require observance of reasonable rules and regulations; 
to select and terminate teachers; to discipline and discharge 
teachers for cause and to contract or subcontract any of its work. 
Article IIIA, 1970-71 Agreement, Campbellsport , Wisconsin, p. 2. 
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*'The Associcitlon rccogni?.es that the Board has the resoonslbil ity 
and authority to manage and direct, in behalf cf the public, all the 
operations and activities of the school corporation to the full extent 
authorized by law.,. It is recognized that, except as exp:;essly limit- 
ed by this agreement, the Board shall retain whatever rights and author- 
ity are necessary for it to effectively carry out its functions and 
responsibilities under the laws of the state of Indiana, These rights 
include thi following: 

7. To contract out for goods and services.** 

Article IV, 1970-71 Agreement , Highland, Indiana, p. A. 

These clauses did not emerge from negotiations over contracting 
out. Rather they were simply copied from existing contracts with a strong 
management rights clause. And in general, although teacher organizations 
will be submitting more proposals on performance contracting, especially if 
and when perfonjiance contracting increases, there is a surprising lack of 
activity relating to contracting out at the bargaining table. 

There are at least two reasons for the inactivity. One is lhat 
relatively few teacher organizations with bargaining rights are currently 
affected directly by performance contracts. It would have been useful to 
interview organization leaders in each district where both collective bar- 
gaining and performance contracting are realities, to see what stance the 
organizations plan to take on the subject at future bargaining sessions. 

Another reason calls for some comment. In Michigan, where 
teacher collective bargaining is more advanced than any other state, the 
Michigan Education Association and its affiliates appear to be taking a 
moderate, albeit confused, position on performance contracting. One reason 



t 






1S3 






is a recent decision of the Michigan Employement Relations Coimnlssion in a 
case involving highway employees. In effect, the commission ruled that it 
was an unfair labor practice for a county highway commission to contract out 
work which could be done by members of the bargaining unit without f:^rst 
bargaining on the subject. Teacher organization leaders in Michigan appar- 
ently believe that there is no need at this time for contractual protections 
relating to performance contracts. In their view, school boards are legally 
precluded from contracting out the normal work of teachers until the boards 
have bargained over such action. Such bargaining Is expected to provide the 
organizations with adequate opportunity to take protective measures. Regard- 
less of whether this interpretation Is sound, It will obviously be easier to 
Introduce performance contracts where teacheis do not have the protection of 
legislation according teachers bargaining rights. Hopefully, performance 
contracts will not be funded on this basis, tempting as It may be to do so 
in the short run. 
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‘I*. rorons idfir oJ 

T('rict)(*r or^’.'in I nation lenders have been «^.tronf'ly cri Lici:<c(l for 
their opposition to the OMO projects in perfonnaTicc contracting,. Such opnosi- 
lion allegedly lefiecls a narrow intercsi on their part in protecting the 
iiiterests of their conatituents , not an unselfish interest in improving 
education. 

There is clearly some fear on the part of some teachers that per- 
formance contracting would lead to a loss of teacher jobs. This fear is 
more of a reaction to performance contracting propaganda than to the reali- 
ties of its performance. Where are teachers - one teacher even - displaced 
because of the greater productivity or accountability demonstrated by per- 
formance contracting? The apparent inability of the districts to utilize 
performance contracting to point to one such case ^ould be of much greater 
concern than teacher organizations overreacting to the non-disappearance of 
teaching jobs. After the millions spent on performance contracting to im- 
prove the effici(?ncy of an enterprise with over two million employees, it is 
unnerving to discover that not a single teacher job has been eliminated as a 
result. Granted, there are other ways of increasing productivity but the 
absence of any increased productivity this way must be regarded as a dis- 
couraging indication of the success of the program. 

In some respects, it may he just as well. For the sake of dis- 
cussion, assume that performance contracting had demonstrated ways for dis- 
tricts to achieve the same output with far fewer teachers. The question 
would immediately arise as to whose responsibility it would be to absorb the 
loss. The individual teachers? Should school districts pay severance pay 
or retraining allowances as an overhead cost of running a school system? 

Should the states and/or the federal government absorb the costs in some 
way as a matter of public policy? Or should tViere be some mix of these 
possibilities, or others not mentioned? 185 
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I rini iujf. wliat l)ie rip, lit answer is, or even whotKcr there is 

one, but surely the worst answer would be to have the excess teac!iers absorb 
tbc etitirc' cost of thnir unoinp] onnent in whatever wavs they can. Such a 
policy elst‘wlioio v i r lual ] y -puarantces omployoo opposition to changes s/hlch 
could lead to grc-nler productivity , and perhaps also to their displc lemcnt . 
Tn the lonp run, policies which emphasise tiu* need to introduce efficiencies 
while assisting difsplaocd individuals to luect t !ie major burdens of dif^plac :i- 
ment are likely to prove the most effective in facilitating greater produc * 
tivity throughout our economy. 

As asserted previously, performance contracting was unwisely 
advocated in some districts as a job saving technique, theregy frightening 
teachers into premature and wholly unnecessary opposition to it. The fact 
is that v;e have some very recent and very relevant experience on how school 
districts handle opportunities to increase their productivity by releasing 
excess teachers; to put it mildly, the results do not inspire confidence in 
the ability of school management to deal with the problem. 

My reference here is to the situation resulting from the const!- 
tional decisiors prohibiting racial segregation in public education. These 
decisions made it possible lor school districts to save hundreds of millions 
of dollars by eliminating segregated facilities and segregated personnel. 

In fact, a very large numb^T did not and still refuses to do so. In many 
cases, districts were virtually forced by the courts to adopt obvious econo-- 
mics. The most crucial point for present purposes is what happened in the 
area of employii*cnt relations. Thousands cf districts implemented the redue- 
tlons in staff v;icli sueli blatant diserimination that even conservative 
Southern judges were eonntrained to call a halt, or a slowing down, 'n 
some cases. Under those circumstances, It in quite undcrstandabl e vdiy many 
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teachers, while and black, react sensitively to suggestions that staff may 
he reduced, I do not assert that the opposition of teacher organizatio is to 
pcrronuanco coptracis is based upon their fear of discriminatory treatment 
in staff redi.cllons. My point is that if and when performanro contracting 
does point the way to genuine increases in productivity, if it evor does, 
there will still be some basic issues relating to implementation that cann, t 
be dismissed as the self-serving opposition of a vested interest. 

Perhaps one additional observation may be in order. In educa- 
tion, the first casualties of performance contracting are not likely to be 
teachers. Tliey are much more likely to be those professors of education 
responsible for teaching the methods and materials courses. At least, that 
is the view of the Chairman of OE*s National Coomiittee on Program Priorities 
in Teacher Education (Dean Benjamin Rosner of the City University of New York), 
one of our most astute students of teacher education. I am inclined to agree 
with Rosner’ s analysis, disconcerting as it is to the established order. 

Bear in mind that much of perfotmance contracting is really inten- 
sive in-service teacher education. If private companies can contract with 
school districts and make a visible improvement in teacher performance within 
a relatively short period of time, our methods and materials experts, along 
with the student teacher lobby, are going to be confronted by some hard ques- 
tions e.g. , what do we need you for when private enterprise can achieve demon- 
strable improvements which you have been unable to produce? It hasn’t come 
to this yet, but it might. 
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IV. Somo Su^.^gcs t Ions 



One* siip,j»c*st ion is that a number of school districts conduct 
an across the hoard analysis of their programs to ascertain which olcmcmls 
could be contracted out. For example, it seems impractical to me to train 
automotive mechanics in schools, since school facilities and personnel can- 
not keep pace with J\o changes in the automotive industry. Why not pay 
industry to provide the training, since it has by far the better personnel 
and facilities to do this particular job. Furthermore, not only would it 
be less expensive to use facilities which are already in place and used 
extensively, but the fiians providing the training could introduce students 
to the world of work in a more direct and meaningful way. 

Of course, this is not a new idea; in fact, it is already being 
implemented in seme districts. However, unlike the OEO approach to date, 

I see no particular reason to emphasize reading and mathematics, or dis- 
advantaged pupils in contracting out instructional tasks. My thought is to 
provide the local museums, dramatics clubs, symphony orchestras, dance 
societies, and other cultural and recreational a'»encies with an educational 
budget, so that they could carry on instructional programs currently regard- 
ed as school functions. 

As a long time tennis player, T have long been concerned about 
the small number of black children who get oppoi'tunities to play and become 
proficient in this sport. There are hundreds of Arthur Ashe's out there, 
if we could only provide the opportunities. Wliy not give children an option, 
wlicreby they can take tennis or golf lessons from private instructors at 
public expense, instead of attending regular gym or physical education 
classes? Such an approach might not result in "performance contracts" 
but in contracts for services rendered. Regardless, I think the average 
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tennis professional is better able to teach tennis than the average physical 
education teacher, who is seldom a specialist in this particular sport. 

In any case, I would like to emphasize two aspects of this sug- 
gestion, One is that controversies over performance contracts have tended 
to obscure the possibilities of less complicated contracts for instructional 
services. It is premature to say that performance contracts cannot be drawn 
up for instruction in music, art, dance, tennis, golf, or what have you, My 
guess is that they can be, Hov/ever, they are very unlikely to be drawn up 
properly in individual districts, especially smaller ones, since the R and D 
expense in drawing up the contracts v/ould be prohibitive. We need model 
performance contracts that can be used, with perhaps some modifications, by 
many districts. It should be emphasized, however, that the contracting out 
of some services should not be dependent upon the possibility of a performance 
contract per se. Performance contracts may eventually develop out of con- 
tracts for services, especially if there is dissatisfaction with the services 
rendered. At the same time, we should continue to explore the possibilities 
of contracted ser voices that are short of performance contracts. 

My second point may be crucial to the future of performance con- 
* tracting. In the long run, some forms of performance contracts are bound to 
generate opposition from teacher organizations. If a contractor can instruct 
as effectively using technology or paraprof essional s instead of teachers, it 
is unrealistic to expect the teachers concerned to applaud the fact, however 
beneficial you and I may think it is. 

The problem here is that performance co*itracting propaganda has 
frightened teachers when it was completely unnecessary and counterproductive 
to do so, even assuming for the sake of argument that performance contracts 
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will quickly demonstrate that we can do the job as well or better with fewer 
teachers. As arp^iiod elsewhere in this paper, education is inefflci :nt, and 
shows little increase in productivity partly because teachers have no direct 
stake in greater productivity. Clearly, one of the basic policy issues to 
be resolved is whether we are going to give teachers a direct stake in 
greater productivity, or whether we are going to discuss alleged or real 
gains in terms of getting nd of teachers or other tenns which generate 
opposition instead of support. The principle should be that management can 
introduce efficiencies but that excessed employees shall have employment 
security, perhaps in new capacities or after some sort of training. A xunda- 
meatal issue here is who should pay for such employment security - the local 
district, the state, or the federal government. As we move toward greater 
state and federal funding, perhaps these agencies will support a larger share 
of these costs. 



ment in the contracts. At present, when performance contracting is intro- 
duced, it often encounters tea :her opposition but generates no interest group 
support in the community. This is especially true where the contractor is 
an outside firm. The case would be much different if contracting were imple- 
mented along the lines suggested above. An imposing number of local organi- 
zations would have a direct and immediate stake in a contract for educational 
services. This may create some new problems while solving the one under dis- 
cussion, but the options ought to be clear. At least, the suggested appro^ ^h 
may be as viable in some districts as those currently supported. Certainly, 
there are enough districts with competent leadership ready, willing, and able 
to try it out. 



The crucial point, however, is the strategy of comnunity involve- 



ISO 



31 



Let me now suggest mother contracting pos*?ibility with basic 
implications for employment relations. I tend to agree with the view that 
secondary schooJs are playing custodial roics more than they should. Some 
of our New York City schools do not even play a custodial role very well; 
in some schools, only 25 to 50 per cent out of the register attend o^ any 
given day, and it is a different group from day to day. Unfortunately, our 
diagnosis, and hence our prescription, for this state of affairs is fiulty. 

Wf do not adequateJy relate the diagnosis to the facilities or to the loca- 
tion of schools, hence we arc. not realistic about the measures needed to 
solve the problem. 

In May, a high school senior is likely to be in class from 8:30 to 
3:00, five days a week. In September as a college f man, he is likely to 
be in class about 15 hours a week. The college schedule permits a student 
to go to the library, work, get something to eat, watch TV, and so on with- 
out direct supervision. Because students are in class much less, it is 
easier to avoid custodial functions while they are in class. 

To illustrate the nigh school problem, suppose a student wants to 
take a private music lesson or see a doctor during his study period. Ordi- 
narily, this cannot be done, because the student would lose toe much time 
travelling betv;een classes. Suppose, however, that when w^ built new schools, 
we also constructed commercial facilities across the street. These commer- 
cial facilities could be rented to a wide assortment of specialists, educa- 
tional, medical, and otherwise. Private teachers of music, dance, and art 
would be located in the complex. Pediatricians and dentists specializing in 
children's problems could also be there. That is, such personnel would be 
there if there was such a facility and if it vras kno\m that pupils could 
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school lor priv.'ilo Icsbons or sorvlccs during, Llic day. The punll who 
f.an’t ^C!t Hpfria I Ixtfd InsirucLion herausu he doesn’t want to miss school or 
be absent from work, or because he lacks convenient transportation would no 
lonp,cr lose these opportunities. Similarly, many other students would benefit 
from the opportunities which could dc provided this way. 

Those sup,gcstions raise a question concerning the OEO projects. 
Should they be vi'ewed as experiments in educating the disadvantaged, or as 
educational experiments with promise for any level or subject matter? As a 
matter of logic, the experiments can be viewed as either or both. Strategi- 
cally, however, the emphasis is crucial. In view of OEO*s mission, it was 
inevitable that its projects were oriented to the disadvantaged. In my 
opinion, this approach was motivated by a sincere desire to educate the dis- 
advantaged effectively. The approach made good sense from some other stand- 
points as well, but in retrospect, it may have been a mistake. For one thing, 
it tends to tie performance contracting to the disadvantaged and to lend cre- 
dence to the charges that perfoinnance contracts per se are a copout on the 
part of public officials with responsibility for educating the disadvantaged. 
More importantly, it has tended to limit an idea with extremely broad ramifi*- 
cations. Consider the following language from a bill intended to authorize 
performance contracting In the state of New York: 

36. **In its discretion, during each of the school years 19/1-72, 

1972-73, 1973-74, to enter into performance guarantee contracts with 
private contractors for the purpose of experimentation to increase 
the achievement level of students in grades kindergarten through 
twelve who have not achieved predetermined goals in reading and 
mathematics under existing methods presently employed by public 
schools. 






I’roposcd Surintc liill 573-B, Cal. No. A19, prcfilod January 6, 1971, 
Introduced by Si;nator Thomas Lavorne, Chairman, Standinf* Committee 
on Kdeoatlon, Senate of the State of New York. 

New York education law has been Interpreted to restrict perfor- 
mance contracting to certain situations specified my law. It may be that 
Senator Lavorne, who is an astute politician as well as a dedicated legis- 
lator, felt that it was necessary to limit perfermano.o contracting If there 
was to be any chance of expanding Its eventual use In the state. Even In 
such case, however. It was probably a tactical mistake to start v;lth such 
a limited approach. 



Mv own view Is that the greatest benefit of performance con- 
tracting Is its tliriist toward the formitlntion of educational objectives 
and their analysis in terms of costs in a practical way. Tlie foilowing 
quotation seems as appropriate today as wlien I first wrote it ir 1959 : 

**The problem ... is that the teachers have failed to establish 
sets of intermediate objectives which would clarify how they propose 
to fulfill the general objectives of education. The discussion of 
objectives is usually concerned wltli the general ones, but this is 
not where the problem lies. It lies in establishing consistent, 
defensible, and attainable intermediate objectives which can serve 
as the basis for evaluating the progress made by the orofesslon.** 
Myron Lleberman, The Future of Public Education 
(Chicago: University of Chicago Press, I960), p. 21. 

Let n.e conclude, however, by emphasizing two points concerning 
objectives. The first is that they are primarily a management resnonsi- 
bility. Scliool management, not teacher organizations, bears the primary 
responsibility for the absence of practical objectives in education. 

Secondly, it should be stressed that the nroblem of settlig and 
adjusting defensible objectives is not peculiar to education. It is, in 
fact, a pervasive problem of government at all levels. As one of the 
country's most brilliant urbanologists asserted recently: 

know of at least a hundred cities ;:hat have had studies made of 
trends in their local economy and other basic life-support systems. 
Perhaps another hundred cities can be found that have attempted to 
establish goals. But 1 have yet to find a city that tries to spell 
out in clear, operational terms just wluit life could be like If the 



city Lh.il irio.'i Lo -inc’ 1 out In cJcnr, oprrationnl terms just what life’ 
could be like If the city could achieve its objcctivea and reverse 
undeslral)le trends or reinforce desirable ones. We expend our cnorples 
on diagnosis and prescription and never gel around to describing how 
the results would look, feel, and act. lu short, we have no clear 
vis ion of the good life to which people can relate effort, sacrifice. 
Costs, and benefits. 

I.eo A. Molinaro, "Truths and Consequences for Older Cities," 

.Saturday Review , May 15, 1971, p. 30. 

My hope is that performance contracting will constitute a major 
step toward solving the problem delineated by Molinaro. If our cdticatlunal 
efforts are successful, what will pupils know, and what will they be able 
to do? If performance contracting can move us toward raising and answer- 
ing these questions in a practical way, it will be a constructive develop- 
ment, regardless of its shortcomings on other g..ounds. 



N.B. This pape*" is submitted for discussion, not as a public state- 
me.nt of the author's views on the issues discussed herein. For 
this reason, dissemination or reproduction should be limited to 
the Belmont House conferees and other appropriate parties desig- 
nated by the conference chairman. No quotations from this paper, 
or attribution of views in it to the author, are authorized witli- 
out permission lu writing from the author. 
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Ouch tlont 

A few years ap,o Dlnekstonc Indicated that he thought that teaching to 
the tent was a good Instructional technique; that In fact, tests were 
proxy variables and that, as Hyler had said, tests were often the same as 
behavioral objectives, so that It made a lot of sense to teach to the testi 
I thought I read in your paper that you thought this was a bad instructional 
technique. Is that correct? 

Answer; 

I made quite a pitch in here about the importance of building a struc- 
ture of knowledge ~ which some people call understanding — and it seems 
to me that if you are willing to settle for rote responses to very pat ques- 
tions, then you are seriously distorting the educational process. Now I 
would like to prepare students to answer questions like the question on the 
test. If I have enough of them, I don't mind letting stvidents see them in 
advance. I am saying it's best you read up on all of the questions because 
a test will be drawn from these 1000 questions. If you know enough about 
the subject, I don't think you are going to be able to commit these to 
memory just by rote; and if you know all of them, you know practically every- 
thing I am trying to teach. Teaching the test, if it is part of the larger 
context of knowledge in which one is going to deal, is not really unethical. 

* * * 

The line is between rote learning and concept /understanding. If you 
have 100 students responding to 100 test items, you have 10,000 responses, 
and I would bet that on the average, less than 100 of these responses (taking 
all those students together) are the result of rote learning in a typical 
examination — that almost always an individual reflects and thinks and con- 
siders alternatives before he gives an answer. Rote learning in education, 

I think, is greatly overestimated — the amount of it that goes on. I am not 
disturbed about the amount of rote learning there is in school; I am concerned 
about incomplete learning, but I would distinguish that from rote learning. 
There are very few things that you can set down and say, "Now I have just got 
to remember this; I'll remember it for purposes of the test; that is all I 
need to do — I don't understand it, but I'll remember this string of words." 
If this does take place, I think the teacher ought to be censured for having 
taught in that way or urged students to try to learn in that way. There are 
a few things that are learned by rote, but they are a small minority of all 
the learning that goes on. 



* * * 

The rote learning problem is not only a matter of knowledge versus some 
other kind of skill. In performance contracting, it may depend on relative 
emphasis upon different types of curriculum. For Instance, in math, in junior 
high school, you could have a program devoted towards learning math concepts 
or doing mathematical problems that are stated verbally and require you to 
translate what you know into something you really haven't seen before. On the 
other hand, you could simply have kids doing as some of our contractors did 
in the OEO experiment computations — kids at third grade math level taught 
to do presecribed "third grade" computations. Now it is very critical to the 
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contrnctor In that case what kind of test Is used. To give a test that 
measures concepts and applications to a kid who has been doing two column 
additions for the year Is going to be very bad for the contractor. There 
has to be some accommodation between the school and rhe company as to the 
degree of congruence that both parties are going to allow In the contract, 
In terms of testing what the company taught. In the OEO experiment, there 
Is very little test congruence — the company has no knowledge of what 
test Is used. The other extreme Is setting up a test and saying, "okay, we 
are going to use that test at the end of the year — now you teach to It." 
That may be an acceptable way to go. It may be more acceptable to have two 
or three different forms of the test so that you can avoid teaching the 
direct test answers. 



* * * 

I think v;hen people talk about the narrowness of performance contracting, 
they forget that It Is possible to be more narrow than they have as yet con- 
ceived. That Is, you could set up a performance contract In vocabulary and 
computation and test just for Chose quantities. That is a great deal narrower 
than Is presently the case because regardless of what Che contract is or of 
what Is taught, you are testing students on two or three sub-tests In reading 
or math, rather than one limited measure. 

ik ^ A 

In Chinking about the goodness or badness ot teaching to the test. It Is 
necessary to distinguish both among subjects and among levels. I can't imagine, 
for example, how one could teach spelling and then test the outcomes without 
having Caught the words chat will be encountered on the spelling test. We 
don't expect children Co know the spelling of words whose spelling they have 
not been taught, generally speaking. In Cue lower grades, we have a restricted 
number of words Chat we expect a child to have mastered as sight vocabulary, 
let's say. So I think there are many situations where the instructional pro 
gram, of necessity, exposes the learner to those tests which will be encount- 
ered in Che measuring devices. The risk in chose situations is chat if there 
is knowledge of precisely which items will constitute the content of the test, 
Che teacher will devote greater attention to those items and thus distort the 
Instructional program. As applied to performance contracting, the issue gets 
a little more complicated. V7e say that a virtue of performance contracting 
is making attainment expectations explicit so that the school people and the 
contractors will devoce thif:mselves more fully, comprehensively, and precisely 
to Che desired goals of the contract of Intervention, But if you want a fair 
assessment, Chen Che measuring instruments likewise ought to correspond with 
high fidelity to Che instructional goals. So in the performance contract 
situation, we are impelled Co a closer correspondence between InstrucClonal 
content and the content of the evaluating Instruments. 

* * * 
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How 1 fppl is rtlwnyH mon* Innortant to mo than wliit 1 l.no*. , But thoro 
1 h nothing that a achool can do to help mo fool good anart from treating me 
like a human being by patting mo on the back occasionally and avoiding frus- 
trating me too consistently. Yet there Is nothing that a school can do In 
the cognitive realm that compares In importance with what it can do In the 
affective realm, VJlthout in any way denying the importance of feeling af- 
fected, then, 1 would question whether they should be made the focus of edu- 
cational programs find whctlier we need to write affective objects into a per- 
formance contract. 



« A * 

Certainly if children are different, it is necessary that you establish 
ultimately diffe nt educational goals for them. But I don't think that need 
get in the way of the performance contractor. We heard this morning that one 
of the characteristics of performance contracting is greater individualization 
of instruction. I believe the contractor who is doing his job will do the 
best he can to discern individual learning disabilities and other problems and 
to tailor his intervention to the individual needs, rather than imposing some 
"off-the-shelf" program on all learning alike. 

A * * 

I am very dubious that the concept of growth or gain is going to be a man- 
ageable one in providing a basis for determining how much should be paid. This 
measure of individual gain or growth in reading or arithmetic is an exceedingly 
elusive concept and the more you wrestle with it from a metric standpoint, the 
more frustrating the problem. I can guarantee you that. 

* * * 

1 would like to wind up by underscoring what I think has prompted all the 
performance contractors to resort to norm referenced tests. Those tests had 
the credibility that made them acceptable to school people as measures. It 
seemed to the schools that these tests permitted comparisons of the outcomes 
of contract intervention with the outcomes of ordinary instruction, so they 
adopted them. Given the performance contracts written thus far, I think that 
the grade equivalent system would be much more applicable if periods of inter- 
vention were longer and if schools were willing to ... average gain rather 
than to measure it for individuals. 

* * * 

I thought you made an extremely important point in what you called the 
dynamic toward continuous improvement. I have used the word iterative change: 
you repeat something, you learn something from it, and you try to do a little 
better. And that's an extremely important aspect of this whole performance 
contracting notion. Performance contracting requires measurement of things 
that are difficult to measure, but each time around, you generate some cumul- 
ative information on the basis of which you learn something for next time. 

As long as you actually do get Information and try to do a little better next 
time, you are way ahead. Sometimes in these experiments, you do a one shot 



experiment and you look at the test reaults and they're either ^ood or bad. 
But they arc not teat results that give you information about how to do 
better next time, and I think that concern spills over into the OEO report. 
You do something for one year, you give the test results, and people make 
decisions about the test results. And that can be misleading unless you put 
information in that report about some other criteria than these scores, 
criteria related to the dynamic of improvement. 

* * * 

Given continued performance contracting, there are people whose jobs 
are threatened in the near future — the professors of methods and materials. 
What is striking is that they sit back and look at performance contracting 
very much from the outside, as though they are spectators; but if a private 
firm can come in and increase the number of teachers who know how to do the 
job in a relatively short time, we are going to start asking some hard ques- 
tions about what these professors are doing. So I am struck by the lack of 
any relationship or concern with us on their part. 

* * * 

I referred to this case in Michigan where the Michigan Employment Rela- 
tions Commission ruled that the Highway Department had to bargain over the 
contractlng-out of work that could be done by members of the bargaining unit. 
Now that's not to say that they can't do it, but only that they must bargain 
before they do it. And apparently, some of the educational leaders in Mich- 
igan are relying on this ruling, feeling that the school districts can't do 
anything without collective bargaining, which would give teachers enough 
notice to block anything, if they feel that they want to block it. 

* * * 

My general line is that responsible evaluation always Involves looking 
for side effects, but these have nothing to do with the program's objectives 
except to define those that weren't objectives. And if you realize that, 
then you must face the fact that side effects may be the dominant factors in 
your overall evaluation. Once you have faced that fact, you realize that 
objectives are just one of the program's variables. They are the ones the 
program was aimed at; if it is a perfect project, you will achieve only 
those objectives and some good side effects. But there aren't any perfect 
projects. So it is perfectly legitimate for an evaluator of these early 
performance contracting projects to state that overall some of them did a 
very good job — not only did they produce some gain in what happened to be 
their objectives, but they produced some incidental gains and no apparent 
bad side effects. Or an evaluator might say that there were bad side effects 
on teacher relations, and that may cancel satisfactory attainment of objec- 
tives. 



* * * 

The only interesting question is whether there are some versions of 
performance contracting that pay. You should not be focusing on overall 
performances at all. You should only be focusing on the best performances. 
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Now there were one or two projects that, on the evidence we have seen, did 
well. And you arc swamping their performance with all the inferior perfor- 
mances. That is absolutely inappropriate, and it is a disaster if we bury 
this movement or even start getting highly sceptical about it because the 
moan performance of performance contractors is bad. That's not Interesting 
in any way. The crucial test of educational innovations is that they work 
overall and in the long run. For purposeu of time, we unfortunately encour- 
aged over-simplified and over-brief evaluations. But education will not 
progress if innovations must pass, and also if they only have to oass, such 
tests. Performance contracting cannot be responsbily evaluated by looking 
only at gains on contract parameters, and certainly not by looking only at 
the specified measures of these parameters. If we also look at measures 
such as the quality of students and the quality of student life and learning 
along with a truly realistic cost analysis, as well as careful studies of 
measurement errors and regression effects, and longer term effects, both in 
terms of retained student perforijiance and the stability of the Innovations, 
then I think we must conclude that schools now can identify some performance 
contractors who arc a very gooc. bet for narticular needs and situations. 

Also, we urgently need to continue this family of experiments, including its 
turnkey aspects and other contracting out procedures, so that we can develop 
better evaluations, ‘ ^tter management procedures, and better contracts, before 
we can allow the whole thing to be decently burled. 
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IV 



Cone lufl lonn 



This section consists of two parts. The first is an elaboration 
of the overall conclusic^ns voiced by Ll»c Chalman at the conclusion of 
the conference. The second is a collection of Individual summaries 
submitted by conferciue participants some nontlis after tbe proceedings; 
this collection represents the considered judgment of the participants. 

A . Chairma n * s Summary 

Five major conclusions seen to liavc emerged from tlic conference: 

1. At this time, the precise impact of pcrfomance contracting 
on student achievement is unclear, and it may remain so for a period of 
years, or at least until the major technical and operational problems 
of student performance measurement arc resolved. Yet the performance 
contracting approacli seems to have exciting potential for school 
system innovation, more rational educational management and resource 
allocation, clearer and more persistent setting of objectives, and the 
development of more useful performance measures. Performance contracting 
ultimately may prove important for strictly non-pedagogical reasons. 

2. Any comprnhensive or summary evaluations of performance 
contracting at this tine and in the future must recognize its diverse 
applications and benefits by employing a range of measures in addition 

to those related directly to the objectives of student achievement. Such 
evaluation must consider not only classroom effectiveness, but also a 
broad range of effects. In addition, it will be useful to consider 
the impact of perfooiiancc contracting on the rate, content, and effects 
of school system change and to evaluate the ability of performance 
contracting to expand the number of tools and agents that public schools 
can use to achieve their objectives. 

3. The private sector probably does have much to contribute to 
public education, especially in training programs. Yet it is not 
certain that pcrfomance contracting is the best way to elicit and imple- 
ment this contribution. 

4. Regardless of the Tate of the performance contracting movement, 
we must not lose sight of the larger issue of contracting itself. There 
seems to be ouch potential in some form of contracting for certain public 
education functions, whether or not the contract in question includes 
very tight performance specifications and a variable payment schedule. 

5. Finally, 1*^ assessing the resources and costs involved in a 
performance contracting program, or in any educational program, we 
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fiiiisL l()ok beyond our muul fr.it, le o( r»!ler«nce. We ti.iy h.ive Lo think of 
ediK ;it Ion .1 tr.iiifif orri.it Ion procenn .ipplled to certain raw matrirlalB - 
MtudetiKi - an one of .1 ncrics of nm lal trc.itnentfl. Hence we may become 
lep.i t In.ttcly concerned with Inprovlnp, the qu.nllty of the student before 
lie entern tlie procenf*, which n.iy redin 0 tae level of resources required 
to effi-ct the trnnsfornntlon; ,ind we M.iy consider the effects of 
'stici efiitf 111" education on the need for and cost of successive social 
» r<-atnents. 
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Performance contracting and turnkey operations were originally 
designed to be a catalyst for school system renewal. The criteria for 
assessing their success must include the following: a means by which 

to introduce cost-effective programs in areas such as math and reading; 
a low-risk, low-cost vehicle for experimentation; an opportunity to in- 
crease community involvement in both planning and operations; an oppor- 
tunity to rationalize the collective bargaining processes; a politically 
acceptable and educationally effective means to ensure school system 
integration or to provide "equity of education results" in those situa- 
tions where neighborhood school feelings are strong; and a means for hu- 
manizing the classroom * both teacher and learner. 

As a low-risk, low-cost vehicle for experimentation and catalyst 
for change, performance contracting combined with effective turnkey appli- 
cations has been Judged to be successful. Whether or not it provides 
more cost-effective instruction is less clear. Recent analyses of 
achievement data in the Virginia project indicate that it was much more 
cos t-ef f;5Ctive than originally thought; "homegrown projects" such as 
those in Grand Rapids have been very successful. On the other hand, 
projects implemented with a short lead time constrained by evaluation 
design criteria have posed operational problems indicating only scattered 
success . 

Because of school-firm interface problems, many of the O.E.O proj- 
ects had very little chance of success after the first month or so of 
operations when these problems arose. 

Regarding the other criteria for judging performance contracting, 
it is just too early to tell. There exists a definite need for new evalu- 
ation designs and testing instruments to be developed and implemented in 
long-range projects such as that in Gary, Indiana. 

Performance contracting in its "first generation" form should put 
itself out of business because of its success, spinning off hybrid ver- 
sions — in Dade County', where individual teachers have entered into con- 
tracts with the school board with the possibility of earning a $ 5,000 
bonus; or the Accountability Model in Michigan where school districts 
could receive $200 for every child w^ho achieves specified levels while 
providing the district the flexibility to spend the money any way desired 
and yet encouraging them to search the market place for the most cost- 
effective systems. 

Much of the data presently available for analysis has been contami- 
nated by the controversial nature of performance contracting, which may be 
an implict compliment alone. Never in the histoiry of public education 
have so few groups with so little resources done so mt\ch to frighten un- 
justifiably, I feel, so many educators. 

- Charles Blaschke 
President 

Education Turnkey Systems, Inc. 
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Impassioned controversy has marked the blazing half-life of per- 
formance contracting. Educators at all levels have been quick to throw 
stones or raise banners. But there is little doubt that knowledge and 
education for the moment are at discount in the United States. This 
is the result of the unforeseen consequence of the use of knowledge, and 
not less frequently, of the presumption of knowledge where perhaps none 
existed previously. It becomes difficult to admit to ignorance and easy 
to assume the reliability of information that is anything but reliable. 
An old saying held that ”it is not ignorance that hurt so much as know- 
ing all those things that ain't so.'* The schools have been more than a 
little hurt of late by that phenomenon, and this, too, has deepened the 
sense of under-achievement or even of failure of the schools to serve 
their purpose in our time. 

But it is true that certain costly school programs introduced with 
great expectations a few years ago are not yielding the promise or ex- 
pected results. In fact the entire concept of a clear-cut positive 
cost-quality relationship in education has been called into question by 
recent reports. Increasingly educators are hearing that there first 
must be hard evidence that a proposed program offers the most effec- 
tive solution available and a tangible return to the taxpayers before 
more funds will be put into it. Citizens are saying that the nation is 
not getting as much out of its investment in public education as there 
should be for the dollars expended. Increasingly we hear the charge 
that the gap between educational promise and performance is a function 
of the outpourings of propaganda by the professional education com- 
munity, and the persistent failure of educational leaders to produce 
implicitly promised results. 

Performance contracting is not new in education. Deleterious ef- 
fects of such contracts are described in the C ggadian Educational 
Monthly of 1881. Its current popularity in the United States is respon- 
sive to a demand for greater accountability. The present performance 
contracting programs represent only a fraction of the variety possible. 
This potential for almost infinite variation is the real strength of 
"performance contracting" and the improvement of the educational system 
demands that its potential be fully explored. This suggests that all 
performance contracting must be evaluated from a broad perspective. 

Faults will be discovered in any program, but the attempt must be made 
to determine whether the faults are the result of that program, that 
contract and that contractor, or whether, on the other hand, the faults 
stem from basic defects in the performance contracting concepts. If 
the evaluative effort focuses merely on the amount of the achievement 
gained and the payment the contractor earns, much of the information 
obtainable from the experience will be lost. 

Like any change in the style and substance of operation, this 
most recent venture into performance contracting involves problems for 
education. Tracing the complex and involved interconnections by which 
"inputs" produce "outputs" in an endeavor as largo as public education 
is not the work of amateurs. It is not yet being done successfully in 
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any other area of our national life, save in economics, and there most 
economists would Insist that It Is being done Imperfectly. It Is not 
being done elsewhere because no one really knows how to do It. It Is 
just that most persons who have considered the matter feel It has to 
be done and accordingly someone will have to learn how to do It. Per- 
haps many someones. That, In short. Is what this conference on per- 
formance contracting was all about. 



- George B. Brain 

Dean, College of Education 
Washington State University 
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(Jhile I understand the conditions vhic'i gave rise to the experiment, 

1 question the propriety of school systems contracting with private firms 
to perform the basic function for which schools have been organized and 
supported. Even so, I sense the desperation In some quarters which led 
to such a practice. I have reservations about the narrowness of contract 
objectives but I also have concerns about school administrators and teachers 
who seem Incapable of specifying what they are about. I think measurement 
problems are being Ignored and that they must be dealt with or payment by 
results becomes a hoax. 

Much of this seems negative but I am not completely negative. When I 
visited the contract sites I found many of the teachers and paraprofesslonals 
Interested and apparently effective. I also found that pupils who were In 
contract programs could be rather explicit about what they were doing, how 
they were achieving, and, for the most part, they seemed Interested In the 
program activities. Some of this Interest might be ascribed to the novelty 
of the program or to the mechanical devices used. Perhaps even more Impor- 
tant was the attention now centered on the pupils, possibly for the first 
time In their school careers. 

Despite the limitations of the experiment, we can learn a great deal 
from It. It seems clear that the achievement of youngsters In reading and 
other basic skills Is a concern at all levels of our society. Moreover, If 
that concern appears to be Inadequately dealt with at local and state levels. 
It will get attention at the national level. Again, In the contracting ex- 
perience the Insistence of the larger society on an accounting by one of Its 
social Institutions, the schools. Is exemplified. The focus of contracts on 
ends reminds us that we are concerned about means as well as ends, that both 
process and outcome are of vital Importance to the educational enterprise. 



- Roald F. Campbell 

Fawcett Professor of Educational Administration 
Ohio State University 
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The survival and, in most cases, the expansion In 1971-72 of pro- 
grams derived from the performance contracting efforts of last year 
support my belief that performance contracting has facilitated the 
Introduction of needed changes In the Instructional process. Specific- 
ally, classroom atmospheres are more salutary and Instruction Is more 
appropriate to students' needs and Interests. The emphasis of perfor- 
mance contracting on student learning was also highly desirable and has 
helped focus attention where It belongs. 

An apparent disadvantage of performance contracting. Its dependence 
on norm-referenced tests for measuring program effectiveness. Is also 
having beneficial effects. The Inadequacy of such tests for the task 
was so strongly highlighted by the programs of 1970-71 that valuable new 
efforts to construct more suitable Instruments have been stimulated. 

Unfortunately, the future of performance contracting Is not prom- 
ising. Its failure may derive from two misconceptions on the part of 
both contractors and school personnel. First, there was an expectation 
that there are quick solutions to the long-standing problems of the 
educationally disadvantaged. Second, despite their Innovative thrusts, 
the performance contracting programs may still have been too much In 
the Institutional mold, especially In the way they used people and time. 
Much more radical changes may well be needed to make education of the 
disadvantaged effective. 



Polly Carpenter 
Senior Staff Member 
The Rand Corporation 
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If the difficulties encountered by some pupils in learning were due 
mainly to the incompetence or poor motivation of their teachers, and if 
the achievements desired of them could be defined clearly and completely, 
and assessed categorically as attained or not attained, then performance 
contracting would offer a promising alternative to conventional schooling. 
But the educational situations in which these conditions prevail are not 
frequently encountered. Hence it is unreasonable to expect performance 
contracting to yield substantial educational benefits. 

The difficulties of validly assessing the effectiveness of perform- 
ance contracting in practical school situations are formidable. Some re- 
ports may overstate, some may understate its actual effectiveness. Exper- 
ience over several years is likely to provide better evidence than a one- 
shot evaluation of the initial trial. 

I do not expect to find In 1975 that more than a very small propor- 
tion (1 or 2%) of the instruction in public schools is being offered under 
performance contract . 



- Robert L. Ebel 

Professor of Education and Psychology 
Michigan State University 
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Experience with performance contracting during 1970-71 indicates 
clearly that Jt was not an immediate solution to the achievement prob- 
lems of compensatory education. Some possible reasons for this are: 

(1) the contractors did not have effective programs; (2) one year is 
not enough time to make definitive judgment on the effectiveness of 
performance contracting in raising student achievements, especially in 
light of the developmental nature of all such programs; (3) the instru- 
ment used to measure what students learned may not have been sensitive 
to what students needed to learn and to what some contractors tried to 
teach; (4) students may not have fulfilled our expectation that they 
try as hard as they can to do as well as they can on the tests we ad- 
minister to them. 

Even though cognitive gains were disappointing, there are strong 
indications that performance contracting was effective in introducing 
new staffing patterns, new roles for teachers, students and parents, 
and new technology, into school systems traditionally resistant to 
innovation. A good measure of this is the way In which teachers them- 
selves, at 3 number of former performance-contracting sites, have shown 
Interest and persistence in continuing the techniques, materials and 
activities originally Introduced by the contractor. 

It seems likely that performance contracting will continue on a 
limited scale for well-defined purposes. Some smaller, less-known 
companies will see performance guarantees as a good way to "break into 
the market." Some districts will see performance contracting as a 
good way to do research and development with new methodology and 
materials, while holding suppliers accountable for results. Vocational 
education is a promising area because of well-defined skill levels and 
performance norms. Finally, we may see much more of "Internal" 
contracting in which the district contracts with teachers or teachers' 
associations for the achievement of specific performance objectives. 



Edward B. Classman 
Office of Education 
> Department of Health, 
Education, and Welfare 
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The concept of performance contracting appears to offer a way to 
effect an Improvement In the management of education by; 

o Acting as a change agent In the use of resources (teachers, 
equipment, materials and time) 

o Supporting changes In the organization of education (Instruc- 
tion, classroom, school and district) 

o Focusing attention on the need for better means of measuring 
program outcome and cost. 

Performance contracts have permitted the LEA to explore In a systema- 
tic manner alternative uses of equipment and materials. These Instruc- 
tional materials are not new to most districts; the way In which they 
are used Is new. Additionally, the LEA has to make an explicit 
guarantee of Its own — to Insure a specified number of Instructional 
hours. An hour of reading Instruction becomes. In fact an hour of 
reading Instruction for the student. 

The greater Individualization of Instruction and the effec Ive 
Use of paraprofesslonals has supported the feasibility of higher 
student-adult ratios as a way of changing the organization of the 
educational process. At the school and district level of organization, 
performance contracting has highlighted the benefit of a program focus 
In achieving an Increased understanding of the resources required to 
produce specific outcomes. This may well hasten the demise of manage- 
ment by geographic location or grade-level and promote a management 
focus at the classroom level. 

The most significant, and widely-acknowledged. Impact of perfor- 
mance contracting Is the powerful Impetus It has given to the efforts 
to develop better means of measuring program outcome and cost. This 
measurement Improvement, almost alone, should have a far-reaching 
Impact on the management of the resources of education. 



Sue A. Haggart 
Senior Staff Member 
The Rand Corporation 



LL is tiangcrou.4 to preLund to draw concjusioius now about pcrfonaaiu'c 
contracting in inlucation given (uiv small store of experience. Never- 
theless, two observations appear warranted. Perforraance contracting is 
not the holy grail of teaching; neither is it the handwriting on the 
wall signifying the end o^ the educationists* bureaucracy. 

The promise it may hold for public education will not even be 
tested, much less fulfilled, unless experiments with the technique meet 
the following conditions: 

1. The allocation of pre-experimental time and resources 
to achieve aggreement among school district participants concerning the 
contract and its implementation. 

2. Careful attention must be paid to mechanisms developed 
to facilitate the process of integration and internalization of innovation 
in the school. 

3. Comprehensive evaluation using multi-dimensional 
measures of performance contracting over time are needed. Additional 
measures of achievement, ones different from those traditionally used, 
are needed, as well as the examination of dimensions concerning other 
than academic learning, e.g., organizational climate, self-image and 
sense of power. 

4. Impacts such as contagion, stress, or rejection of the 
innovation by other units of the organization also need to be understood. 

5. Finally, evaluation designs should leave room for the. 
discovery of un-anticipated consequences. 

For nov;, however, the studies on performance contracting tell us 
more about what is wrong with federal policy making than about the 
performance contracts. Federal policy choices influence the selection 
of goals and priorities by other educational authorities. So long as 
federal policy in education continues to search for the holy grail of 
instant results via narrowly defined efficiency measures, innovations 
are doomed to trivial outcomes. Until innovations are given a reasonable 
allocation of time between the issuance of capital RFP*s and agency decisions, 
as well as time for planning and start up, the innovation effort will be 
so much wasted resources. Our study of innovation and change will be 
characterized by ever learning and never coming to knowledge. 



“Laurence lannaconne 
University of California 
at Riverside 
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For approximately the first twelve months following the introduction 
of guaranteed performance contracting in Texarkana, hundreds of school 
administrators Jumped on the innovation bandwagon and still thousands of 
others talked about the eventuality of implementing a guaranteed perform- 
ance contract in their school system. Now thirty months later, the AiTier- 
Ican public, particularly the educators, have become fully aware that 
guaranteed performance contracting is not the panacea that they thought 
It might have been. Allegations of teaching for the test and education 
being subsumed by the industrial complex have taken their toll. Most cri- 
tical, however, is the fact that the guaranteed performance contract simply 
has not created the results in terms of educational achievement, that is, 
pupil output, that it initially was supposed to have. What guaranteed 
performance contracting has done, however, is to create a mechanism for 
both accountability and management. 

The ever increasing demands of communities to be Involved in educational 
decision making have resulted in cries for school districts to make public 
their student achievement. Perhaps making education accountable for its 
output has been the major value of guaranteed performance contracting. I 
would contend, however, that most, educatorio have been aware of output all 
along. They may not have owned up to their shortcomings, but they have 
always been very much aware of the difficulties they faced. The most sig- 
nificant value of guaranteed performance contracting, I believe, has been 
that in order to Implement a guaranteed performance contract educators have 
had to take a long hard look at process. It is the changes to the process 
of education that will in the long run have the greatest Impact on output. 

Less than half of the number of companies engaged in performance con- 
tracts last year are similarly engaged during this school year. The future 
of guaranteed performance contracting can, therefore, be considered rather 
dim. However, the fact that guaranteed performance contracting is being 
phased out of the educational spotlight is not wholly bad. Performance 
contracting has served its puirpose. It has succeeded in priming the pump 
of accountability, monitoring, and management and by doing so has created 
a group of educators more aware of process and output in that system which 
we call education. 



- Michael H. Kean 

Assistant to the Superintendent 
School District of Philadelphia 
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My attitude toward performance contracting on the basis of the 
discussions at the Joint AERA-AASA conference, and Information 
and reflection, both prior to and subsequent to the conference, 
Includes the following elements: 

1. Performance contracting and achievement Improvement . The weight 
of evidence to late seems to be that performance-contract Inter- 
vention does not consistently lead to better-than-average achieve- 
ment gains. At best the verdict on performance contracting as a 
producer of better-than-average achievement has to be "not proven." 
Data thus far available do not permit any confident assertions 
with respect to amount of achievement gain per Instructional dollar, 
but It seems altogether unlikely that the performance-contract mode 
of Instruction will loom as more efficient than conventional modes 
In this respect. 

2. Other presumed benefits of performance contracting . Given the dis- 
couraging achievement gain outcomes associated with performance 
contracts, there has been a disposition on the part of advocates 

of this mode of Intervention to say that even In the absence of 
conspicuous Improvements In achievements gains, performance con- 
tracting, nevertheless, had other significant values - e.g. In- 
creased flexibility, encouragement of open classroom approaches 
innovative approaches, etc. These claims are worthy of attention, 
though It must be declared that the evidence In support of them 
Is not highly persuasive; but even If one accepts these declarations. 
It Is Important to realize that the basis for discussion has been 
significantly altered: we are now asked to think of performance con- 
tracting as Impacting process variables, not product, which Is 
quite contrary to the bases on which performance contracting was 
originally most forcefully advocated. 

3. "Performance contracting" as a variable . Consideration of the 
results of the early performance contracting studies heightened 
the realization that "performance contracting" per se Is a poor 
experimental variable. The variation from one performance contract 
experiment to another In effectiveness forces our attention to the 
quite variant modes of Intervention that have been utilized In 
several performance contracting situations. It seems clear that 

It Is simplistic to talk of the effect of "performance contracting" 
rather than to talk of the effects of the specific types of Inter- 
vention that were the genuine experimental variables In the several 
projects. Methods, materials at.iJ teacher competence, as well as 
degree of outside Intervention, all varied from one project to 
another; It seems clear that the impact of these variables, sep- 
arately and In combination, need be evaluated separately from the 
performance contracts' mode of Intervention before any assertions 
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can be made as to the special contilbutlons or utility of the per- 
formance contract mode. 

Measurement aspects of performance contract activity . Everyone 
recognizes that the measurement processes in the conduct and assess- 
ment of performance contracted intervention leave much to be desired. 
There is specia.. need for more, rigorous attention to instrument 
selection in relation to the specific goals of the contract inter- 
vention, including most particularly metric properties of the in- 
struments chosen in relation to considerations of reliability of 
growth measures: very much more careful planning of the measure- 
ment aspects of performance contract interventions with respect to 
such items of testing, training of examiners, security of instru- 
ments, etc.; far more advance attention to analysis of results in 
relation to performance contract goals; and far more advance atten- 
tion to modes of collecting and reporting test information in rela- 
tion to the terms of the contract. There is widespread feeling 
that most available standardized tests have significant limitations 
with respect to the fulfillment of measurement demands of perform- 
ance contract situatr'jns, particularly when the duration of the 
performance contract intervention is limited - say, less than a 
year. Nevertheless, even with existing instruments it seems possi- 
ble, at least in this observer's opinion, to do with careful plan- 
ning a considerably improved measurement Job in relation to per- 
formance contract programs than has typically been the case to date. 

I think that the results of the OEO study and the attendant publici- 
ty may have, in fact, been the death knell of performance contracting. 
I would regard this as unfortunate, at least in the sense of imply- 
ing a premature Judgment of the efficacy of this mode of interven- 
tion. A full and fair evaluation of this arrangement for the provi- 
sion of instructional support would require, in my Judgment, a 
longer period, a more careful specification of outcomes, identi- 
fication of ways of measuring outcomes and ways of relating perform- 
ance contract outcomes to control or anticipated outcomes. I sus- 
pect that it is unlikely that performance contracts meeting these 
desiderata will be entered into in the foreseeable future. It 
became clear in the course of the AASA-AERA conference, if it had 
not been before, that the motivation prompting many school adminis- 
trators to enter into performance contract arrangements had little 
to do with hoped for better-than-average increases in achievement— 
that administrators seized upon performance contract arrangements 
for other (though quite possibly no less worthy) purposes, which. 

In fact, may have been achieved, to some extent at least, through 
the performance contract Interventions. An evaluation of perform- 
ance contracting ought certainly recognize these perhaps hidden 
agenda, and suggest the utilities of performance contracting for 
the advancement of certain school management purposes, even In 
the absence of dramatic contributions to Improved achievement. 




- Roger T. Lennon 

Senior Vice President 
Harcourt Brace Jovanovlch, Inc. 



My belief is that performance contracting may have value for Amer- 
ican education. However, It has been so badly managed thus far that the 
idea Is unlikely to receive the kind of thoughtful consideration it deserves, 
at leant In the immediate future. It is especially unfortunate that per- 
formance contracting fiascos have discouraged Interest In subcontracting, 
an idea whl':h may have even greater relevance to our educational needs. 

In my opinion, the opposition to performance contracting by teacher 
organizations was the natural and to be expected result of the ineptness 
of those who espoused and sought to Implement the idea. Vlhlle I regret 
that experience to date with performance contracts has been overgeneral- 
Ized to mean that the concept has no utility foi /merlcan education, I 
believe also that opposition to the performance contracts in existence 
or recently completed would have been greater if full information about 
their genesis and inadequate substantive rationale had been widely available. 



- Myron Lieberman 

Director, Office of Program 
Development and Administrati- n 
(Teacher Education) 

City University of New York 



We are at that point in the life of an innovation not unlike the 
day .ter a romantic honeymoon when the delightful young wife steps from 
her automobile on the way to the grocery and is killed by a truck. We 
know that the romance is over, that the bliss cannot repeat Itself; but 
we've had so little experience with the thing that we neither know what 
it meant or whether there will ever by another event like it. A kind 
of ennui or paralysis has taken over. 

Remember the principle of llghter-than-alr vehicles, which was 
coupled with long-distance air travel In the form of the dirigible? 

Jus** about the ..Ime this innovation seemed promising, the tragedy of 
the Hlndenberg disaster cut short the career of the dirigible. Yet, 
in other forms, we still have llghter-than-alr as well as long-cllstr; ce 
air travel. By analogy we are In the weeks after the Hlndenberg crashed. 
It Is impossible to predict what will happen, and somewhat difficult to 
characterize where we've been. Will we build more dirigibles? A dif- 
ferent kind of aircraft? Crucify dirigible manufacturers? Abanr.on air 
tavel? Has our experience anything much to tell us about what we might 
do next? I think some of the elements of performance contracting, as 
we've known it since 1969, will reappear in several guises. Vouchers 
is one such. But whether OEO's blowing up of the Hlndenberg will kill 
our dirigible, I just don't know. 



- James L. Mecklenburger 
Phi Delta Kappa 
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T\\e full potential of performance contracting has not been realized 
, in my opinion, warrants further careful study. 

Although a majority of research studies report negligible pupil 
achievement gains, many research experts emphasize the lack of appropri- 
ate, valid evaluation procedures and instruments for these studies. 
Evaluators and administrators involved in the programs appear to agree 
that performance contracting offers opportunities for low-risk, low-cost 
innovation and change. 

It seems to me that reports have minimized positive ’’by-product** 
values because they were not primary goals. For instance, some programs 
have resulted in decreased pupil dropout rates, but this fact has been 
overshadow^:d by emphasis on low gains in reading and math. 

I believe that performance contracting requires more time, with 
expertly devised evaluation, before valid judgments can be finalized. 

The enclosed resolution adopted by the AASA membership at their annual 
convention in February substantiates this position. 



- Paul B. Salmon 
Execu t ive Secre tary 
Ameiican Association of 
School Administrators 



The pressures for change in public schools have never been greater. 
Taxpayers want more for fewer dollars, while school personnel bargain for 
increased wages. Governmental agencies and minority groups demand that 
minority-group children receive equal (not necessarily identical) educa- 
tional opportunities; parents and community groups want to be involved in 
the planning and operations of schools; and pressures for accountability 
are multifaceted and real. Performance contracting has been suggested as 
one feasible solution to many of these problems. 

We believe that performance contracting allows schools to experiment 
with and validate new learning systems with low risk and costs. We do 
not believe it has demonstrated total cost savings in overall school bud- 
gets, although it may do so in specific areas. We support the application 
of the concept by school districts with adequate evaluation so long as it 
is perceived as a means for effecting positive change. 



- Resolution adopted by the 

American Association of School Administrators 
Annual Convention 
February, 1972 
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My overall view of performance contracting (PC) is; 

1. On the data presented at Belmont, the best contractor was doing 
very well, and the deviation from the mean was significant enough 
to make it highly probable that we now have identified a procedure 
which can provide massive gains on basic skills, by contrast with 
the usual approaches. 

2. On the- data I have seen since, it appears extremely likely that 
several other contractors also produced substantial gains, a fact 
that was masked earlier by statistical and test artefacts. 

3. Recent data on retention strongly suggests these gains are not 
transient; the Philadelphia tests by IDEA in fact suggest that 
retention is much better than with the usual procedures. 

A. None of the data so far establishes long-term retention of substan- 
tial gains and it is essential we follow-up. 

5. None of the data so far establishes cost-effectiveness advantages, 
but the main issue for many parents is effectiveness even at higher 
costs . 

6. The reactions of most Belmont participants suggested that they 
thought the appropriate standards for judging educational innova- 
tions are that big gains must be shown the first year by the median 
performer. You don't flunk the whole math class because the median 
student flunks. Education is in bad repute and bad condition not 
because successful procedures are unknown but because they are dis- 
carded as unfashionable rather than being developed and disseminated 
and quality-controlled. 

7. One of the most important features of PC is that it puts severe 
pressure on the contractor to look at the cost-effectiveness of the 
various procedures he might use. The educational scene has been 
b^dly lacking in agents on that reinforcement schedule until now. 

Of course, this will produce some abuses which we shall have to 
learn to control by improved contracts, auditing and evaluation. 

But the present system performs so badly in teaching basic skills, 
by comparison x^ith schools in several other countries, that it is 
hard to avoid regarding it as a far worse abuse. 



- Michael Scriven 

Department of Philosophy 
University of California at Berkeley 
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The results of the Office of Economic Opportunity's Performance Con- 
tract Experiment Indicate that participating private firms operating under 
performance contracts In school year 1970-71 did not perform significantly 
better than traditional school systems. These findings are quite similar 
to those reported by the Rand Corporation In Its survey of nonexperlmental 
performance contract programs. 

OEO commented on Its findings that "it Is clear that there Is no evi- 
dence to support a massive move to utilize performance contracting for re- 
medial education. School dlstrclts should be skeptical of extravagant 
cJalms for the concept." 

This does not mean performance contracting Is dead but does Indicate 
need of considerable development. Measurement techniques, Incentive 
clauses and contract administration procedures need considerable refine- 
ment. The capabilities of private firms themselves may also need devel- 
opment. 

Performance contracting might best be considered as an Infant con- 
cept, that will change In the future. Varied applications of It are pos- 
sible Including Internal contracting by teachers, Incentive contracts for 
administrators or nonacademic programs. The range of possible programs 
should not be underestimated. Nevertheless, such programs should be con- 
sidered high-risk because of their undeveloped stage and pursued carefully. 

Performance contracting has shown Itself to be a simple concept, 
whose value was Initially overestimated, that Is complex In execution. 

Its long range potential Is presently unclear. 



Giarles B. Stalford 

Office of Economic Opportunity 



Performance contracting must still be considered in the realm of 
unproven Innovations. It Is Innovative not because It Is a new Idea 
for groups to contract for and pay for services only after those services 
have been delivered In accordance with specifications, but because Its 
application In education calls for pajrment to trainers only after learners 
have learned In accordance with learning specifications. This Innovative 
application still must be considered unproven since It Is not at all clear 
that by following the practice of performance contracting students learn 
better than they otherwise would. Certainly before this Innovative prac- 
tice can be proven, It needs to be worked out technically better than 
it has so far. Thus, a variety of measurements and statistical problems 
must be solved before learning specifications associated with performance 
contracting can be adequately stated and tested. Before these technical 
problems are worked out. It would seem to me Inadvisable for school 
districts to be overly zealous In entering Into performance contracting 
with firms who are anxious to sell such services on other than a cost- 
sharing experimental basis. 



- Daniel L. Stufflebeam 

Director, Evaluation Center 
College of Education 
Ohio State University 
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The last chapter in performance contracting has not been written. 
New research Is being conducted, the results of which refute many of 
the preliminary conclusions reported In the 1971-72 OEO experiment. 
These new findings lead one to question the conclusions reached In 
the OEO experiment, If not In fact the total project design as well 
as the final treatment of data. 

In the OEO experiment, preliminary conclusions were drawn by 
assuming all performance contracting companies had to be successful 
in all grades in order for performance contracting to be successful. 
This was not the design of the original project. No consideration 
was given In the final treatment of the data to reflect Its Impact on 
different age level students, different performance contracting pro- 
grams or different student populations. 

Several other Independent studies have shown that though students 
in some of the experimental programs did not achieve one grade level 
gain in achievement, they more than doubled their output from previous 
years' experiences. 

One such Independent research project conducted by Dr. Joan 
Webster^ In the Grand Rapids Schools refutes many of the OEO findings. 
Dr. Webster's research of a cost-effective model used two school age 
groupings - early elementary and junior high students - and Included 
six reading programs; a control group using the systemwide adopted 
reading program; three different performance contracting companies' 
programs; a publisher's program; and a traditional remedial reading 
program. 

Dr. Webster found no significant difference In reading achieve- 
ment gain In any of the programs studied In the lower grades. However, 
she did find one performance contracting program significantly less 
costly than all the other programs. In the junior high programs, only 
the performance contracting companies' programs experienced a year's 
gain In reading achievement while the control group and the remedial 
reading groups achieved approximately one-third as much. The cost 
effectiveness of the programs again were significantly In favor of 
both performance contractors' programs; with one contractor's program 
being considerably less expensive than the other. 

One might conclude from the results of this rather comprehensive 
study that there are significantly different cost effective reading 
programs for underachieving children offered by some contractors. 
Further, the results would tend to Indicate that performance contract- 
ing programs In reading may not be as adaptable and effective for 
early elementary children as for older children. 




^published Doctoral Thesis, Joan M. Webster, "Cost Effective 
Analysis of Six Reading Programs In the Grand Rapids Public Schools," 
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In view of the above ^ it is this writer's opinion that performance 
contracting still shows promise in raising the achievement level of 
some students but cannot be considered the one solution for ail the 
learning problems in education. 



Norman P, Weinheimer 
Executive Director 
Michigan School Boards 
Association 
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Orlnioiil lloiisr 



Klkriil^o. Mar) IniMl 



ru 0 (;u AM 



Tliiirsda), DetTUiber 9 




RrcnkfasI 
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American .VKsociatioii of Scliool Adiiiiiiislralors 

Ricliard A. Dcrsliimcr 
Executive Officer, A ERA 

2. Cliairnian 8 Opening Remarks 
Donald M. Levine 

The Ontario Insliliite for Studies in Education 

Introduction of Conference Participants and Staff 
Purposes of the ('.onfercnce 
Organization of the Conference 
Output of the Conference 

3. Commissioned Paper: 

Major rmhtnns in i*frjnnnnnrr Cf)iUrartin^ fnr l\tlnration: l/i InimJurlion 

Donald M. I.evitie 

Assistant ProfesM>r and Director 

Studies in Educational Policy, Plaiuiin^, and Performance 
The Ontario Insliliilc for Studios in Education 



1. Coiimiissioned Paper: 

. 1/1 hvalnaiinii of iWfnnwtnvo (^nnirorlitif: fnr llh.il 

Polly Car|>eiiter 
Senior Staff Mcmlier 
The RAM) O/rporalion 

2. Discussion 



10:1 5 a.iii. 
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10:45 a.ni. 



Plenary Session 



12:30 p.ni. 



Lunch 
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2:30 |Mii. 
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4:30 p.m. 
5:00 p.m. 



6:15 p.m. 
7:00 p.m. 
8:.30 p.in. 







Kra(liii|»n — ('.oimiiivMdnvvl Papovs: 

^nnir hithlriits in l'w>d/fir t 'flnrntiniwt iWfornutnrn 

KoIhmI i.. 

Profi»>M)r of Kdiiratlfm ami Psyrholo;;y 
Mu'lii^an SlaU’ I niwoiiy 

hroi:rnni (.o.sl \ihi(\ >h in fjinnititnwl rimming 

Sue A. Ila;^nrl 
Senior Slaff .MrniLKT 
Tlie HAND Corporal ion 

Small Group Di^'iis.sion Hilli R. Kbel, S.A. Ilag^arl 

(Meeting of llic groups nilh each atillior separately for one hour.) 

Group 1 — Chairman: ^l^ehael Striven 

Department of Philosophy 
t'niversily of California at Berkeley 

Group 2 — (.hairtnaii: Daniel Sttiffleheaiii 
Evaluation Center 
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Nashville, Tennessee 

Group 2 — Chainnan: George N. Smith 
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Porfornmnrv Inuani U hni liostiil: \n I'.xanunnlnni ofSomo of Iho 
Probhniji in Oulroino ^Irasnroniout 

Selma J. Mushkin 

Professor of Ecoiioinics and Director 
Public Service.*? Laln^ratoiy 
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I'l . hiMriiilii'i I II 



IIMMhi.m. 
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f 
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11:00 a.m. 



12:30 p.m. 
1 :30 p.in. 
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3:00 p.m. 




MrtMkf.ist 

Itr.idin;: - (iumini.Hsionrti Tajirr: / //m’/iZ Ki lnlton^ ( ndrr l*rrfornianrr 

( oiilrth h, ^l. l/irhrniuM 

Plenary 

1 . Cliairinan's Opoiiiii;: RoiiiarkA 

2. Comnii»iuiicd Paper: 

l.nijfhtx im nt lirLtlion% I tuIrrPrtfornuiHrr Cmitnirts 
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Ohio Slate Lniversity 

Group 2 — □lairman: (}eor^c Drain 

Dean, College of Education 
Wasliiiivion State University 

Liinrli 

Plenary Session: N'iews from I lie .\dininistrators 

Topic: The rnn fhuturr) Implirntions of Porfornmneo Cnnlractirif: 

for hducntion: I Mona^nnrnt l\ialuntioii 

Giairman: Gordon McAiidre^^ 

Snperinlendent of Schools 
Gary, Indiana 

(x)ffee 

Plenary Session 

1. General Diseiis.«ion 

2. Cliairinnn's Summary' 

Donald M, Ixvinc 

Ontario Institute for Studies in ndiicatioii 

3. Concliidiii" Remarks 
Adjourn 



-51- 

Z30 



Piiui oiiMVN(.i. ( oN'ii: \ i)\{ i;i)i cvnov 
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uun n 

Tlmis(la) . DimtiiiIkt 
2:.I0 |Mii. — i:30 i).m/ nullmo.n 
( .iKiiniiiiii: Dr. Mirlurl Srii\rii 
2::U)^:i:;iO D . Itnlinl l.. Kl»r| 
3:30 - 4:30 Viis5 .Sue A. Ila^prl 
5:00 “ 6:15: H;jilroonf 
Chairman; Dr. T.. C. S(imb<*rt 

Mr. Cliarlfft L. lUasrlike 
Dr. Koald Cani|il)<*ti 
Dr. Richard nmliinier 
Dr. Slanley tiam 
[)r. .Nolan tstra 
Dr. Robert Glaser 
Mr. Michael II. i\ean 
Dr. Donald M. l.c\ine 
Dr. Gordon ^lc Viidren 
Dr. Paul A. Milier 
Dr. William Rtiriiell 
Dr. Paul Salmon 
Dr. Michael Striven 
Dr. E. C. Stinibert 
Mr. Bro l Ual 



GKOl Pll 

Thursday, Drcciiibrr 9 

2:30 — 4;.30 p.n*.: Coiifereiirc Room 

riuiniMii: Dr. Daniel Stuffleheaiii 

2:30 ~ 3:30 Vli^i^Sue A. Ila««ar( 
3:t30 - 4:30 p.in. Dr. Knlierl I.. Khel 
5:00 — 6:15: Cmiierenre Room 
Chairman: Dr. Geor;;e N. Smith 

Dr. George Brain 
Mrs. Polly Carpenter 
Dr. William Elicna 
Mias Martha Gable 
Dr. Edward Glasstnan 
Dr. Laurence lannaecone 
Dr. Roger Lennon 
Dr. Myron Liebermaii 
Dr. James Mtcklcnbiirger 
Dr. Setiiia Muslikin 
Dr. Terry Saario 
Dr. George N. Sniitli 
Mr. Charles H. Stalford 
Dr. , Norman Weinhciiner 
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I)r. (f<»rdoii Mr Andrew 
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Dr, D.oiiel Stnrneheaiii 
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Polly CuriNMilor nMfivrd Imt |{urlirlorV riiiii lamle, in Kdiiralioii IVoiii llu* Ohio Slalr 

I iiivcr.sily. and her Masler's d<‘pr<»e in Mulheniulies from I he same iiiiiversily. Prior lo joining 
I he Kami (^orporalioii in I9ri5, slie lun^lil ul Ohio Slale and in Ihe |>nblie s<'liools of Florida. 



\l Uaiid she ori{i;iiuilK Horked on |irohlems of deeision-inukin» diiriti*; warlinie, Inil in 1907 
she eliun^ed her area of inleresi lo ediiealioii. Siiiee llien she has |)ailiei|irted in a sliidy on 
|)lanniii(]; ediiralional rhan^e in (iolonibia for Ml), has MTVed on a ( niversily of ralifornia 
eoininillee lo plan uhi\ersilN pro;'rains in arehileeliire and nrhan and re;;ional plannin|]i;. and has 
Horked wilh oilier iih'IiiImms oi Ihe Kand slaff lo fonniilale a program hnd<;e|in^ sysleni for 
loeal srhool disi riels. Her Htirk in Ihis area is ineorporaled in **Pro«jrani Knd^elin^ for Seliool 
Disiriel Planning, lo Ik* available Iroin hihiraltmtaf I rrlinolft^y l*fihliratiftns in 1972. She has 
als<» Invii aelive in developing Ihe lerliiiH|iies of eosl-effeelivene.ss analysis for a.ssi.slanee in 
decisioiHiiakin^ at Ihe slale and loeal levels and hass 4 *veral piibliealions in ihisarea. She is now 
doeninenling some ol Rand*s work on Ihe evalnalion of performanee eonlraeliii^ in pnblie 
ednealion for the Deparlmenl of Ilealih, Kduculion. and Welfare. 

(.nrrenlly, Polly heads Ihe Kand leain developing looks for iiisirneliuual s-yslein de.si«rn under 
Vir Koree sponsorship. She also heads llie Kdnealiuii group in Hand's Sysleni Sciences 
Deparlmenl. Her res«‘areh inleresls eeuler on ihe def inil ion <if objiT lives in ednealion, measures 
of elieeliveiies.s. I he design of inslriirlioiial sysh*nis. and Hie inslriiclional prore.ss. 
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KOUKKT L, KKKL 



KoIhtI L. IVoiVssor Kdiiralioii and Ps\rliolo^y a| Mirlii;;raii Stair I'liiirrsil). is a 

>|NTialisl ill llir field of rdiiralioiial inoasiimiiriits and rCM^ardi. Hr rainr to Mirlii^an Stair 
I iiivrrsily in from llir Kdiiralioiial Trstiii" Srrvirrs of IViiirrloiu Nrw JrrM*v >vlirn* lir 

liad Imh ‘11 a \ irr IVrsidriil ^illi ^riirral rrspoiisiliilily for Irst drvrlo|Miiriil. slalisliral anal) sis. 
and llir dirrriioii of Irsllii" |H'o»raiiis. 



li^dorr Joiiiiiiv ihr Kdnralioiuil Trsliii« Sriurr staff in 1957, l)i\ Klirl was Professor of 
Kdiiralioii and Dirrrior of tlir Kxaiiiiiialion Srr\irr and the Hiirraii of Kdiiralioiial Urs^Mrrli at 
the Stale I iiiversil) ol Iona. In Hie years iM'lueen 1952 and 1944 lie s<*rved as a lii|*li school 
teacher of seieiiee. Me rirceived a B. A. degn*e from the I niversily of Norlherii Iowa in 1952. 
an M.A. in 1956, and a Pli,l), in 1947 from the I’niversily of Iowa. 



Dr. Kind is a inemlN*r of the \mei*iean P.sryholopeal Assoeialion, the Amerieaii Vssoeialion for 
the Advanremeiil ol Srieiu*e, and the P.syehomelrie Smdely. He is a past president of the 
National (.oiineil on Measiirenienis in Kdiiealion. and is presidenl-Heel of the Anieriean 
Kdiiealioiul Kesearrh Assotdalioii. He is author or eo-aiilhor of s<‘veral hooks on measnriii" 
ediiralional aehievenieni and ol niinieroiis prolessional articles on the appliealioti of 
ediiealional t<‘sts. He has reenill) eilited the I’oiirlh Kdilion of the KneyelofHMiia of Kdiiealional 
Kes4»arrh. 
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Mis> ( ui uilli a llarlirlor ol Srinirr ilr^ivr in Uiisiiir.s.s Sdiiiiiiislialioii fioni llu* 

1 o)‘ r.aliloriiia. I.os Vii^rlrs, in Siiirr that liinr slir has t'oiiliiiiird lirr shulirs in 

rriiiininir.s «*i|(;iiirri in^. ami iMilllirliialir.s. 

Mis«i lla^f^ati JoiihmI llir Kami slat! in 1955. lirr rarly shnlirs worr roii(*rnnMl with nmiiiiainl. 
ronliol and roniiiiuiiicalimi.s: m'iimii* and inlrllifvtMirr sy.slnns; air drlVnsr sy.slniis: and ollirr 
rli‘rlroiiir*nrirnlrd s^slrins. Sin* has M*r\<‘d as a ronsiillaiil on sliidirs ol* similar sysl<‘nis lor (hr 
Oflirr ol‘ Mm* S«*rn*lai‘) ol DofriiM* and Ihr loiiil (lliiffs ol’ Slafl*. In I96f>. sin* M*rvrd as a 
ronsiihani lo Tin* I' ord h oiindalioii on Mndt jMoposal lor I In* Uroad rasters' Non|irol'il Satellite 
S> stein. 

Dnrine 1968, she nas the |iro|eel leader ol Kaiid's stud) on the iis<* ol‘ |irotfrani*biid^et iii^ 
teehni(|iies as a nn'lhod lor |dannin;); elementary and secondary ednralioii. This ellort uas part 
ol the (ialilornia*uide project known as Operation PK8 (Prepariii|:; Kdiicalional rianner.s). 

She is the project leader lor Kand's work with llie Research rorporation ol the \ss<N'ialion ol* 
School Unsiiiess oflicials. The K(7 ASIU) project involves the conceptual desi^fii ol* a National 
Plamiine - I'ro^rainniin^ — Uiid^etini; — K\ahialion System (PIMIKS) lor education. She is als<i 
one of the principal staff meiiihers of Rand' • study of |H*rforinance coiilractini; in edncalion 
spon.sored hy tin' V.ssistanI Secretary 's office of IIKW. 

Mis.s lla^^art's enrrent efforts arc focn.sed on analysis for educational plannin*; and on the 
application of the teidniiqnes of pro^ani biid^etiii^ in the areas of education, health and other 
public systems. She is tlie Domestic Studies group leader for the Resource Analysis Department. 

Mi.ss llaggart's publit atioiis im lnde: 

f*Tui:ntm nu tmil fur .\ hool Ihsirirl ZV/o/ /////". S. \. Ilaggart, M.R. 

(*.ar|)eiiter. The Rand (‘.orjioration, P—4051, February I%9. 

\imfv>is o/ l.thntil nptnil l*rtt^nints UHhin a l^rtt^nnn S, \. Ilaggnrt, M.R. 

Cai|H*iiter, The Rand ('.orporation, P — fl95. SepteiiilN*r 1969. 

I >rr**lnftinu n t^npisnnii Ittulurlini: \\ sh‘in as an loZ in ZVa/oi/n^' t li^fwr h thnutl am, S. \. Ilaggart. 
The Rand ('.orporation, P—4252. November 1969. 

I*nt^nnn I nr St latnf hislra l l*lannin^: (.tmrrftis aial I />/>//<•/// S. A. Ilaggart, 

J. A. Dei Ro.ssi, S. M. Rarro, M. R. (*.arp(*iiter, M. L. Rapp, The Rand (iorporalion. 
RM— 61 16— R(*., NoveinlM*r 1969. 

\nalvzinL* I hr / sr nf t ta'hnninjjy In / ft^ratlr I'ahauit inn m u />cre/o/>/n^' (.'nuntrw M. R. 

Carpenter, L.C. Chester, It,S. Dordick, and S..A. ilaggart. The Rand Corporation, RM— 61 79— 
RC. March 1970. 

OtsI I flrrlivr |m//vM.v fnr tjlia nlinnal nanninn, M.R. Car|M*nter, S. V. Ilaggart, The Rand 
Corporation. P—4527, ^la^ch 1970. 

I hr l*rni:ram .^Inirlnrinf: \sfa'rl nj ITU fnr hhaalinn, S.A. Ilaggart, 1‘he Rand Cor|Hiratioii. 
P-4456. March 1971. 

h'njrr! H 'i, San Jtfsr { alifiprnia: f.rntnaliipn nf lirsntls ami Ihnvlnptnrnl o/ I Cnsl IZo/ZcZ. 



M.L. Rapp, M.R. Carpenter, S.A, llaggart, S.il. Landa, G.C. Sumner, The Rand Corporation. 
R-672— SJS, April 1971. 




DON \U) M. U:\IM: 



Donald M. Lrviiir is Vssislaiil ProtVssor at TIm' Ontario Insliliilc tor SIikIn's in ^duration. 
Hr is also Dirrrlor of a major rrs4*arrli |)ro;:ram al lln‘ liisliliilr riilillrd. "Siinfirs m i flnruliniml 
l^nhry. tnni f*t‘rfint}\tn\rv'\ Dr. I.rviiir's niaiii inirrr.sis arr in rditralional poliry 

analysis, planning, and iiiana^rinnil. 



Dr. Lrviiir has had roiisidrralilr rxjirririirr in Ihr planning: and inipiriiiriilalioii of a widr 
raii^e of |M)liry, sysleni and o|)rralions rrs4Nirrh sindirs in iiidiisir). Hrforr roiiiiii" lo I'hr 
Ontario inslilule lor .Sindirs in Kdnralioii in l%9. Dr. Lrviiir srrvrd as a senior systriiis 
analyst for Ihr Syslrins Vnaiysi.s and Krs4'arrh ('.orporatioii. Oainhrid;:(*. Massarhn.M^tlst I'hr 
Xrrox Corporalion. Korhr.sirr. \ru ^orkand thr (lohiinhia |{roadrastiii<: S>striii in Nrw York 
(.ily. Krcrnlly he has focused on Ihr iim* of program hiid^rliiij; in school systems, and has 
lec lured frequently for the Nalioiial Academy of School Kxecu lives on educational planning 
and program budget iiig. 1 1 is recent Ht>rk includes "^tnnturin^f l*rn*irnm I nal\\si,\ for KdiiniUonaf 
Kr.srarrh" (The RAM) Oorporalioii, P— July 1971) which was delivered lo the Joint 
Study (iroup on Military Resource MIocalion Methodology (jSOOMR AM), on \m 

hirntums m hinnititouif 26lh Meeting, 1970 and*' \r}nrrin^ ttotami'il linphmrntat hm 

o/ l^rofrram for h'.dunition'* (The RAM) (.orporalioii, P-467r>, Ortolicr 1971). In 

addition. Dr. Levine has roinpleled Tfw lio{r of \nntysi.s in which is included in the 

Rrp<irl ol rile Second National (lonferencr on PPRKS in Rdncalion rondneted bv the Rrs<Nirch 
(.orporation ol the Association ol .School Rnsinrss Officials, During 1968-70, he was a nieinbi*r 
of the Danforth honndaliun Study (Harvard ream) of Large (aty School Systems, Dr. Levine 
has been a (.onsnilani to the National Program Budgeting Project for Kdnratioii (1970-71) 
and The New ) ork Stale (^oiniiiission on the Oiiaiily, (,ost and hinajicing of Kleiiientary and 
Secondary Lducalion (1971), This year he participated in the design and stdeclioii «>f a thirty 
man policy team for (ieniral Mortgage and Housing, a Canadian Crown Cor|>oralioii. 



Dr. Levine serves as a Considlaiil lo the RAM) Coqioialion, I’he Mas.saelmsells Higher 
Rdncalion Racililies Commiss^ii, and the liileiides Research Ronndalioii. 



Dr. Levine rweived a ILA. from Darlmonlh College ( I960). He also holds the B.A. (1962) and 
the M.A. |()\on., (I966)| Irom Oxford 1 niversity. He received his doctorate from Harvard 
liiiversily in 1970, He is a ineinlxM' of the American Rdiicalional ReM*arch \s.s(M’ialioii and The 
Operations Research Socudy of \nierica (ORS A), 
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>n KON UKUKKMAN 



Mmoii l.irlM'i'iiKm DirrrliM' ol* liir Ofl'lrr of IVo^raiii i)r%'rlo|>inrtil (Trarlior Kdiiralioii) al 
IIm’ ( jl> I iitxMsil) of Nru ^ork. Ills s|MM’ial arra of iiilrrosl is rollorlivo iK'^otiulioiis in 
<Mlii(‘alloii. Hr lias Ih^mi a DiriMioi' of llir Pin Drila Ka|)|>aii Nalioiiul liisliliilcs on (!ollnii\r 
Nr^oliulions in Kiliiralion at Harvard I nivn.sily and al I hr ( ni\rrsll\ of (ialiforniu. Hr has 
also hrrn a ronsnitani on rolirrilvr nr^olialions lo Ihr National Srliool Hoards Assorialion as 
v^rll as a nr^ollnlor lor a i^idr varirly of m’IiooI hoards and Irarhrr or;|[ani/.alions inrhidiii;!’ thr 
Philadripina Srhoot Hisirirk Ihr I niird I’Vdrralion of Traclirrs, and Ihr Nrw York Hoard of 
hdnrallon. Hrrrntly hr has artrd as a roiisiillani lo Ihr Nrw ^ ork Slair (ioniniission on Ihr 
(,)nalil\. (.osl and I'hunirin^ ol Kirinrnlary and Srrondary Kdiu*alion and lo ihr Nrw ^ ork 
(illy ( oniinlllrr on \rrounlahilit\ . 

Prior lo joiniiijj; thr farnll y ol (!il\ t nivrrsily. Dr. Lirhrnnan si'rvrd as Profrssor of Kdnralion al 
Hhodr Island (!ollrjj;r, IVovidrnrr, Khodr Island from l%.'M%9. Hr holds a Harlirlor of 
Srirnrr dr^^nr in Law (Jiiiir, Dili) and in Kduralion (Aii^^ish 1948) from thr rni\rrsil> of 
Minnrsota. In addition, hr rrrri\rd llir M.A. ( Aii^nsl, 1950) and Ph.l). (Lrhruary. 1952) 
from liir t nivrrsily of Illinois al I rbana. 

Dr. Lirhrnnan is widriv known asu rrilir of Ainrriraii rdnralioii. Hr is ihr anilior of t-.tlm'dliun 
n /Vo/r*,vs/o// tPrriilirr-llall. 1956) and Thr I'nhirr of l^uhlir Tdiiraimn ( t'liivn-sily of 
r.hira^o Prrss, I960). In addition, hr was ro-anllior of ('olirrinr \ntohulitnts fttr Tr(irhrt\\ 
and liroiliitiis of ( (tifrrinr \r^ttiuih nis fttr Trarlirrs (Kaiid MrNalK, I960 and 1967, 
rrs|H‘rlivrl> ). His rrrrni work iiirlinirs M'l vinp as "iirsl rdilor for Phi Drila Ku|>|>aii in llirii 
DrrrinlHM'. 1970 issiir rnlilird f \rtulr.\ im \rroiintnhiiilv. His work as Dirrrior of I hr 
National (‘.oid'rmirr on (iollrrlivr Nr<^olialioiis in Hi^lirr Kdnralion a|)|N*arrd in llir 1971 
volmnr ol ihr WiMonsiii Law Krvirw. His arlirir rnlilird f^rofrssors f nitr! apprarrd in 
llar|H*r\s Ma<];a/aiir (Orlohrr. 1971). Hr was piirsl rdilor this yrai of llir Sprruil Srriion on 
"Hnsinrss on Ihr I rlian Kronliri : Nrw (.omitiuiiilk's — a .Sp«*ial Is.siir of Saturday Hrvirw" 
(May 15. 1971), Dr. LirlM^riiinn has siTvrd as a Irrinrrr for Ihr \mrriran Kdtiraltnnal Krsrarrh 
\s.soi‘ialion, Ihr \niriiraii \sMN*ialion of Srliool \diiiinislralor.s, Ihr Dniird h'rdrralion of 
rrarlirrs, and llir National Kdnralion A.ssorialioii. 
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SKLM\ MISHKIN 



Si'liiiji Miislikiii is a IVoicssor ol Kroiioiiiirs and Him* lor of I hr INihlir Smirr.s La hoi a lory at 
(fCorm'lo^^ii loKrrsih, LalKiialory llial fosins and riij^a^rs in iiil<*riiiii%<*rsil> sliidirs lo 
assist stairs and loralilirs in program )>olir> anaKsis. Slir rrrrivrd lirr IL \. di'j'nv Hilli honors 
Ironi HrookUii (adh*«r. hrr M,\, from (iolninhia I nivrr.sit). and rii,l). dr»irr from Ihr 
'fiadiiatr l*arutl> ol Socud Srirnrr of Ihr Nr>\ Sriiool for Sorial UrM^arrh. Prior I o roiiiin^ lo 
(fTori’rlomi in 1970, shr >»as a rr.HMrrh profrsMir at Tlir (irorjjr WashinjO^*** Lnivrrsil^ and 
Di'-rrlor ol Ihr Slalr-LiK-al hinaiirrs IVojrrI al Ihr Lnivrrsily, Karlirr, slir was a rrH*arrli 
prolrssor al 1‘lir Johns Hopkins I iiivrrslly. Shr has HTvrd as a I'on.^iiltaiil niaiidy on sorial 
program finanrrs lo man> inirrnalional and frdrral a;;eni’irs inrlndiii^ Ihr ()K(il), WHO, 
P\H(), I NhS(.0, IIKP. I)()l), HI I). OMH, All), and HKW, Hrr rarlirr rinployniriil was with 
Ihr I rhan Insliluir, Ihr \d>isor> (ioiiiiiiission on lnlrr<i;ovrrninriilal Rrlalions, Ihr Puhlir 
Hrallli Srrvirr. OHirr t i Ldiiralion, and more than two drradrs ago slir was (lliirf of tlir 
Division of hiiiaiirial Sliidirs of Ihr Sorial Srriiritv Admiiiistralion. 



Shr has wrilirn rxlrnsivrly on Irdrral graiils-in-aid and inlcrgovrriiinriilal relations, and 
parliripalrd in alinosi rvrr> major roniinission study on inlrrgovrriinirnlal rrlalions, starting 
with the (irovrs Coniinitliv in tlir rarl> forties, Ihr Maiiion, latrr Knrstbaiiiii, r.niiiiiiissioii, 
Ihr joint hrdrral-Slatr Art ion ('.oinniillrr, and Ihr Vdvi.Hiry (ioininission on Intrrgovrrnnirnlal 
Krlalions, 
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